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Abstract 


The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  provides  a  well 
defined,  publicly  available  set  of  methodologies  for  providing  appraisals  relative  to  Capability 
Maturity  Model®  Integration  (CMMI®)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage 
modes,  including  both  internal  process  improvement  and  external  capability  determinations.  With 
the  publication  of  this  handbook,  the  method  is  embodied  in  three  standard  variants  based  on  the 
class  structure  defined  in  the  Appraisal  Requirements  for  CMMI  (ARC  V  1.1).  As  a  set,  the 
SCAMPI  methods  provide  a  variety  of  solutions  to  accommodate  the  needs  of  appraisers  who 
play  a  variety  of  different  roles.  The  internal  change  agent,  the  professional  consultant,  and  the 
external  auditor  all  have  needs  that  lead  to  specific  sets  of  tailoring  decisions  in  the  use  of  process 
appraisals.  Guidance  for  these  needs  is  provided  for  each  applicable  process  description.  This 
document  defines  the  boundaries  of  tailoring  and  provides  guidance  for  the  application  of  the 
SCAMPI  B  and  SCAMPI  C  methods. 
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Overview 


This  document  provides  procedural  information  regarding  the  conduct  of  Standard  CM  Ml 
Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  B  and  SCAMPI  C  appraisals. 

Document  Contents 


Chapter 

Contents 

1.  Introduction 

This  chapter  consists  of  a  brief  overview  of  the  document  and  its 
contents,  an  executive  summary,  and  the  method  overview. 

2.  Process 
Descriptions 

This  chapter  consists  of  the  core  procedural  requirements  and 
guidelines  for  the  conduct  of  SCAMPI  B  and  SCAMPI  C.  This 
chapter  is  intended  to  provide  the  authoritative  definition  of  these 
methods. 

Appendix  A 

Appraisal  Requirements  for  CMMI  (ARC)  Traceability:  SCAMPI 

A,  SCAMPI  B,  SCAMPI  C 

How  to  Use  This  Document 

This  document  serves  as  the  process  definition  of  the  SCAMPI  B  and  SC  AMPI  C  methods.  The 
primary  role  of  the  document  is  to  serve  as  a  reference  volume  during  training  and  field  use  of  the 
methods.  It  is  expected  that  the  first  chapter  of  this  document  may  be  read  by  a  broader  audience, 
while  the  remaining  chapters  are  of  primary  interest  to  team  leaders  and  members  of  teams.  The 
document  incorporates  layout  conventions  derived  from  Information  Mapping™  techniques, 
which  are  intended  to  aid  navigation  and  information  retrieval. 

Bi-Directional  Traceability  of  Requirements 

Most  of  the  formatting  conventions  used  in  the  document  are  self-explanatory;  for  example,  the 
key  words  in  the  left  margin  for  each  block  of  text  identify  the  topic  covered  in  that  block.  In 
Chapter  2,  many  of  these  key  words  are  also  accompanied  by  tags  that  identify  a  specific 
requirement  from  the  Appraisal  Requirements  for  CMMI  (  ARC).  In  combination  with  the  ARC 
Traceability  Appendix,  bi-directional  traceability  is  documented. 


®  Capability  Maturity  Model  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

SM  SCAMPI,  SCAMPI  Lead  Appraiser,  and  CMM  Integration  are  service  marks  of  Carnegie  Mellon 
University. 
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Executive  Summary 


The  following  is  a  high-level  overview  of  the  key  concepts  covered  in  this  handbook.  It  is 
provided  to  support  the  information  needs  of  executive  decision  makers,  as  well  as  others  who 
may  want  to  know  about  these  methods  without  knowing  how  to  implement  them. 

Purpose 

The  acronym  “SCAMPI”  stands  for  Standard  CMMI  Appraisal  Method  for  Process  Improvement. 
(CMMI  stands  for  Capability  Maturity  Model  Integration.)  With  the  publication  of  this  handbook, 
the  method  is  embodied  in  three  standard  variants  based  on  the  class  structure  defined  in  the 
Appraisal  Requirements  for  CMMI  (ARC  V  1.1).  As  a  set,  the  SCAMPI  methods  provide  a 
variety  of  solutions  to  accommodate  the  needs  of  appraisers  who  play  a  variety  of  different  roles. 
The  internal  change  agent,  the  professional  consultant,  and  the  external  auditor  all  have  needs  that 
lead  to  specific  sets  of  tailoring  decisions  in  the  use  of  process  appraisals.  Guidance  for  these 
needs  is  provided  for  each  applicable  process  description.  This  document  defines  the  boundaries 
of  tailoring  and  provides  guidance  for  the  application  of  the  SCAMPI  B  and  SCAMPI  C  methods. 

Flow  of  Activity 

An  initial  contact  between  the  potential  appraisal  (team)  leader  and  the  appraisal  sponsor  focuses 
on  the  business  objectives  that  create  a  need  for  the  appraisal.  During  the  planning  process,  the 
needs  of  the  sponsor  are  matched  to  the  type  and  magnitude  of  appraisal.  The  elements  of  the 
organization  to  involve  as  well  as  the  scope  of  the  model  to  examine  are  also  specified  as 
elements  of  a  comprehensive  appraisal  plan.  Next,  appraisal  data  gathering  is  typically 
accomplished  through  interviews  and  a  thorough  review  of  documentation.  Then,  appraisal 
results,  including  strengths  and  weaknesses,  are  based  on  analysis  and  verification  of  information 
which  is  compared  to  relevant  portions  of  a  CMMI  model.  Characterizations  of  individual  model 
practices  are  typically  summarized  using  a  red/yellow/green  scheme,  and  are  reported  along  with 
the  detailed  findings  statements.  Finally,  results  of  the  appraisal  are  documented  and  archived  in 
accordance  with  the  agreements  established  during  planning,  and  required  data  are  submitted  to 
the  CMMI  Steward. 

Core  Concepts 

Five  important  core  concepts  that  appraisal  users  should  be  aware  of — the  SCAMPI  family 
architecture,  objective  evidence,  data  sources,  practice  characterization,  and  appraisal  outputs — 
are  covered  in  this  section.  Important  issues  relating  to  the  appropriate  use  and  reporting  of 
appraisals  can  be  traced  to  these  five  concepts. 
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The  SCAMPI  family  architecture  differentiates  three  classes  of  methods  by  identifying  the 
primary  focus  of  SCAMPI  A,  B,  and  C  as  “institutionalization,”  “deployment,”  and  “approach” 
respectively.  The  SCAMPI  A  method  has  rigorous  standards  for  detailed  data  collection,  and  for 
identification  and  coverage  of  the  organizational  unit.  The  SC  AMPI  B  method  retains  some  of  the 
requirements  for  detailed  data  collection,  but  provides  relaxed  standards  for  sampling  the 
organization.  The  SCAMPI  C  method  has  relaxed  standards  relating  to  evidence  of  usage.  These 
methods  can  form  building  blocks  for  a  progression  of  appraisals  -for  example,  starting  with  a 
SCAMPI  C  reviewing  the  process  descriptions,  then  a  SCAMPI  B  investigating  their  deployment 
to  projects,  finally  leading  to  a  formal  benchmarking  event  focused  on  institutionalization  of  the 
practices  across  the  organization. 

Standards  of  objective  evidence  for  the  SCAMPI  family  distinguish  among  direct  artifacts, 
indirect  artifacts,  and  affirmations.  Individual  items  of  evidence  examined  during  an  appraisal  are 
related  to  the  practices  and  other  components  of  the  CMMI  model  in  use.  Traceability  between 
items  of  evidence  and  the  project  or  organizational  entity  to  which  they  apply  is  also  maintained. 
Using  the  definitions  for  objective  evidence,  we  can  define  Practice  Implementation  Indicators 
(PIIs)  to  manage  the  information  collected  during  an  appraisal.  Documents  or  databases  that 
catalogue  and  map  PIIs  can  be  used  as  an  input  to  the  appraisal  process  or  be  selected  as  an 
output  of  the  process. 

The  data  sources  used  in  appraisals  focus  primarily  on  interviews  with  technical  and  managerial 
staff  as  well  as  a  review  of  documentation.  Presentations  made  by  members  of  the  organization, 
along  with  demonstrations  of  tools,  may  also  be  used  as  data  sources.  In  SCAMPI  C,  information 
is  sought  that  describes  the  approach  taken  (or  planned  for  the  future)  to  implement  practices 
consistent  with  the  intent  of  CMMI.  In  SCAMPI  B,  evidence  of  implementation  (not  just  the 
intended  approach)  must  be  examined,  in  the  form  of  direct  artifacts.  This  more  rigorous  method 
also  requires  the  use  of  interview  data  in  support  of  a  team  consensus  process  to  derive  the 
appraisal  results. 

The  use  of  practice  characterization  is  a  hallmark  of  the  SCAMPI  family.  The  characterization 
scales  are  designed  to  match  the  level  of  conclusiveness  with  the  standards  for  data  collection  and 
analysis  in  each  method.  Characterizations  for  each  practice  in  the  appraisal  scope  can  be 
determined  for  the  organizational  unit,  as  well  as  for  individual  projects  or  other  appropriate 
organizational  entities.  Characterization  and  rating  of  goals  are  not  permitted  in  a  SCAMPI  B  or 
SCAMPI  C  appraisal.  While  the  characterization  scales  are  designed  to  support  planning  of  future 
actions,  they  are  not  legitimate  predictors  of  future  success.  An  effective  approach  to  meeting  the 
intent  of  the  CMMI  goals  may  be  rendered  useless  through  poor  implementation.  Similarly, 
institutionalizing  an  approach  that  works  well  for  only  a  small  segment  of  the  organization  may 
not  be  feasible. 

The  appraisal  outputs  associated  with  each  class  of  method  are  also  matched  to  the 
conclusiveness  supported  by  the  appraisal  process.  The  SCAMPI  C  and  SCAMPI  B  methods 
require  the  generation  of  findings  statements.  Using  a  red/yellow/green  scale,  the  two  methods 
also  support  the  derivation  of  detailed  results  mapped  to  each  model  practice.  In  the  SCAMPI  C, 
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a  characterization  scale  reflecting  the  fidelity  of  the  approach  in  reference  to  the  intent  of  CMMI 
is  available,  but  alternatives  are  permitted  as  well.  In  SCAMPI  B,  the  characterization  reflects  the 
extent  to  which  the  examined  practices,  if  implemented  across  the  organization,  would  likely 
contribute  to  the  satisfaction  of  process  area  goals.  Finally,  an  Appraisal  Disclosure  Statement  is 
required  as  an  output  of  every  SC  AMPI  appraisal.  This  standardized  appraisal  statement  serves  to 
document  an  accurate  description  of  the  result,  and  must  be  submitted  along  with  other  appraisal 
outputs  to  the  CMMI  Steward. 

Rough  Order  of  Magnitude  Estimating 

The  SCAMPI  C  method  can  be  performed  in  a  single  day  by  a  single  (qualified)  individual, 
performing  the  tasks  described  in  this  chapter,  for  a  narrow  scope  of  the  model  and  organization. 
SCAMPI  C  appraisals  with  a  larger  scope,  in  terms  of  the  organization  or  model,  are  performed 
using  an  appraisal  team  working  over  a  number  of  days.  Sometimes,  a  professional  consultant 
will  elect  to  perform  a  SCAMPI  C  chiefly  because  it  can  be  performed  alone,  while  the  SCAMPI 
B  requires  use  of  a  team.  A  SCAMPI  B  used  as  a  dress-rehearsal  for  a  SCAMPI  A,  in  contrast, 
will  often  employ  four  to  eight  team  members  over  a  five-  to  eight-day  period. 

Appraisals  of  medium  to  large  magnitude  are  always  performed  using  an  appraisal  team.  By 
dividing  the  team  into  sub-teams  of  two  or  three  individuals,  the  process  areas  in  the  scope  of  the 
appraisal  can  be  assigned  to  individual  mini-teams  and  the  data  collection  and  analysis  work  can 
be  distributed  efficiently.  A  team  of  six  to  eight  skilled  and  knowledgeable  appraisers  can  perform 
an  appraisal  focused  on  seven  to  fifteen  process  areas  in  five  to  ten  days.  A  SCAMPI  C  is  not 
recommended  for  large  teams  or  appraisals  of  broad  scope,  unless  there  is  ample  justification  for 
the  expense  in  light  of  the  limited  generalizability  of  SCAMPI  C  results. 

Why  Do  Multiple  Appraisals 

A  formal  benchmarking  SCAMPI  A  appraisal  is  rarely  the  most  appropriate  event  to  initiate  a 
program  of  model-based  process  improvement.  As  well,  most  appraisal  sponsors  and  process 
improvement  champions  can  ill  afford  to  wait  for  a  benchmarking  appraisal  to  gain  insight  about 
the  status  of  the  process  improvement  program.  The  set  of  three  SCAMPI  methods  allows  the  use 
of  a  sequence  of  appropriately  tailored  appraisal  events — as  illustrated  in  the  discussion  of  the 
SCAMPI  family  architecture  above — to  meet  a  variety  of  needs. 

Summary 

The  first  step  in  preparing  for  an  appraisal  is  to  contact  an  SEI-authorized  appraisal  (team)  leader. 
The  sponsor  or  a  designee  will  initiate  a  dialog  about  the  objectives  and  key  parameters  of  the 
appraisal.  Based  on  experience  and  the  constraints  of  the  work,  an  SEI-authorized  appraiser  can 
tailor  the  appropriate  method  to  meet  the  needs  of  the  sponsor. 
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Method  Overview 


This  section  covers  material  that  is  key  to  understanding  the  intended  use  and  implementation  of 
the  SCAMPI  B  and  C  methods.  Rather  than  focus  on  detailed  procedures  and  requirements,  this 
section  provides  a  systems  view,  describing  how  the  appraisal  components  fit  together  to  form  a 
coherent  picture.  Key  concepts  associated  with  the  methods  are  described,  with  an  emphasis  on 
the  variety  of  ways  the  requirements  for  these  methods  can  be  met. 

Method  Context 

The  SCAMPI  B  and  C  methods  benefited  from  the  practical  experience  of  numerous  talented  and 
experienced  professionals;  their  contributions  over  the  years  have  made  process  appraisals 
successful.  The  intent  of  the  SEI  Appraisal  Program  is  to  provide  qualified  professionals  with  the 
ability  to  creatively  define  appraisal  solutions  to  satisfy  organizational  and  process  appraisal 
needs. 

The  vast  experience  gathered  from  the  practical  use  of  many  different  appraisal  methods  has 
played  a  substantial  role  in  shaping  SCAMPI  B  and  C.  The  influence  of  the  Software  Process 
Assessment  (SPA)  and  Software  Capability  Evaluation  (SCE),  which  grew  out  of  the  original 
work  by  Watts  Humphrey  as  well  as  subsequent  generations  of  methods,  such  as  the  Capability 
Maturity  Model  Based  Assessment  for  Internal  Process  Improvement  (CBA IPI),  SCE  Version 
3.0,  and  EIA  73 1 .2,  can  be  seen  in  the  definition  of  these  methods.  The  pilot  appraisals  were 
staffed  with  software  and  system  professionals  who  brought  experience  with  many  other  methods 
and  models.  These  included  the  SETs  Interim  Profile,  and  the  suite  of  methods  used  by 
Honeywell,  Intel,  Lockheed  Martin  M&DS,  Motorola,  and  the  large  group  of  acquisition-oriented 
organizations  listed  in  the  Acknowledgements  section. 

The  intent  of  the  SC  AMPI  B  and  C  project  was  to  define  a  robust  framework  that  skilled  and 
knowledgeable  professionals  can  use  to  tailor  an  appraisal  event  to  meet  the  needs  at  hand.  There 
are  many  different  ways  to  satisfy  the  Required  Practices  and  the  Parameters  and  Limits  of  the 
methods.  Thus,  there  is  no  single  (limited)  definition  of  the  set  of  operational  procedures  that 
compose  the  SCAMPI  B  or  SC  AMPI  C  methods.  Rather,  there  is  an  explicit  set  of  requirements 
and  guidance  for  how  to  configure  an  appraisal  event  that  meets  the  needs  of  the  organization  and 
satisfies  the  requirement  for  the  method. 

Method  Objectives  and  Characteristics 

The  CMMI  product  suite  includes  the  Appraisal  Requirements  for  CMMI  (ARC),  which  provides 
a  publicly  visible  architecture  for  the  specification  of  a  variety  of  appraisal  methods  designed  to 
meet  common  requirements.  Key  objectives  for  the  SCAMPI  B  and  C  methods  include 
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•  broad  flexibility  to  define  an  appraisal  event  to  match  business  drivers 

•  selectable  components  that  can  be  assembled  to  create  an  appraisal  event 

•  a  proven,  well  specified,  and  flexible  framework  to  lower  the  risk  and  cost  of  separately 
building  unique  appraisal  solutions 

Documentation  for  the  methods  includes  the  specification  of  Required  Practices,  Parameters  and 
Limits,  and  Guidance.  Role-specific  guidance  sections  also  help  to  illustrate  how  the  methods  can 
satisfy  a  variety  of  needs  that  had  previously  been  met  only  through  use  of  different  appraisal 
methods.  Therefore,  it  is  important  to  understand  that  a  skilled  and  knowledgeable  user  is  needed 
to  design  and  perform  these  appraisals. 

Method  Core  Concepts 

SCAMPI  Family  Architecture 

The  SCAMPI  family  of  appraisal  methods  contains  three  different  appraisal  method  definitions 
(  A,  B,  and  C)  that,  in  combination,  span  the  range  of  typical  appraisal  objectives.  The  tailoring 
and  scaling  permitted  in  each  method  class  is  so  wide  that  the  three  methods  overlap  in  many 
respects;  this  overlap  is  evident  when  the  full  range  of  customized  appraisal  events  and  the 
options  they  support  are  considered  in  detail. 

SCAMPI  B  and  C  are  intended  to  support  focused  individual  applications,  as  well  as  use  in  a 
multi-event,  on-going  program  of  improvement,  acquisition,  or  consultation.  Used  in  a 
progression,  all  three  methods  in  the  SCAMPI  family  can  be  tailored  to  enhance  integration  of 
data  from  one  appraisal  event  to  the  next.  The  SCAMPI  family  is  designed  to  be  interoperable, 
where  the  specified  outputs  of  appraisals  can  serve  as  inputs  to  subsequent  appraisals. 

Pilot  tests  of  the  methods  revealed  that  there  is  a  high-payoff  sequencing  and  tailoring  strategy:  a 
sequence  of  appraisals  focused  on  approach,  then  deployment,  then  institutionalization  is  used  for 
a  single  organizational  unit  (or  a  group  of  organizational  units).  In  this  strategy,  the  first  appraisal 
examines  the  approach  taken  (or  planned)  by  the  organizational  unit  to  implement  practices  that 
satisfy  the  goals  of  the  model.  Later  appraisals  then  examine  the  deployment  of  these  practices 
(newly  deployed  or  long  institutionalized)  to  the  projects  or  other  groups  within  the 
organizational  unit.  Finally,  the  benchmarking  appraisal  (SCAMPI  A)  establishes  a  rating  result  to 
demonstrate  institutionalization  (either  newly  achieved  or  re-verified). 

This  sequence  is  best  illustrated  with  an  example  from  the  B/C  pilots.  One  organization  had  an 
existing  CMM  implementation  and  wanted  to  migrate  to  CMMI.  The  organization  developed 
proposed  changes  to  existing  processes  to  make  them  CMMI-compliant.  In  addition,  new 
processes  were  designed  to  address  the  process  areas  not  found  in  CMM. 

Step  1:  A  SCAMPI  C  was  conducted  on  the  proposed  processes  (approach)  to  determine  their 
fidelity  to  the  model.  The  question  being  addressed  was:  If  these  proposed  processes  are 
implemented,  would  they  satisfy  the  goals  and  meet  the  intent  of  CMMI?  Using  the  data  from  the 
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SCAMPI  C,  the  organization  could  then  make  adjustments  to  these  processes  to  increase  their 
confidence  of  fidelity  to  the  CMMI  model. 

Step  2  l  The  processes  were  implemented  in  pilot  projects.  The  SCAMPI  B  method  could  then  be 
used  to  examine  the  results  of  executing  these  processes  (deployment).  A  key  question  the 
SCAMPI  B  data  could  address  was:  If  the  processes  examined  in  the  pilot  projects  were  broadly 
implemented,  would  they  satisfy  the  goals  and  meet  the  intent  of  CMMI? 

Step  3  l  The  organization  could  then,  with  a  higher  level  of  confidence,  broadly  implement  the 
processes.  A  SCAMPI  A  could  then  be  used  to  validate  institutionalization  and  establish  a 
benchmark  for  the  organization. 

The  utility  of  each  method  in  the  different  phases  of  the  process  improvement/appraisal  life  cycle 
described  above  is  affected  by  the  decisions  made  in  customizing  the  method  for  use.  The 
SCAMPI  C  method  is  well  suited  for  an  appraisal  focused  on  the  approach  taken  to  satisfying  the 
goals  of  CMMI.  Appraisals  that  focus  on  the  deployment  of  processes  across  an  organizational 
unit  are  less  well  supported  by  SCAMPI  C,  where  more  comprehensive  data  collection  standards 
would  be  required  to  better  support  that  focus.  SCAMPI  B,  on  the  other  hand,  is  well  suited  to 
either  approach  or  deployment.  The  SCAMPI  B  method  should  require  relatively  less 
modification  to  accommodate  either  focus  or  even  to  focus  on  both  approach  and  deployment. 
Limited  information  about  institutionalization  may  also  be  obtained  using  SCAMPI  B,  with 
appropriate  data  collection  standards.  Finally,  SCAMPI  A  can  be  tailored  to  accommodate  all 
three  types  of  focus. 

SCAMPI  Family  Data  Structure:  Objective  Evidence 

The  definitions  for  different  kinds  of  objective  evidence  are  given  in  the  glossary  of  the  Method 
Definition  Document  for  SCAMPI  A.  They  read  as  follows: 

Direct  Artifact:  The  tangible  outputs  resulting  directly  from  implementation  of  a  specific  or 
generic  practice.  An  integral  part  of  verifying  practice  implementation.  May  be  explicitly  stated 
or  implied  by  the  practice  statement  or  associated  informative  material  [MDD  method  overview]. 

Indirect  Artifact:  An  artifact  that  is  a  consequence  of  performing  a  specific  or  generic  practice  or 
that  substantiates  its  implementation,  but  which  is  not  the  purpose  for  which  the  practice  is 
performed.  This  indicator  type  is  especially  useful  when  there  may  be  doubts  about  whether  the 
intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but  there  is  no  indication  of  where 
it  came  from,  who  worked  to  develop  it,  or  how  it  is  used)  [MDD  method  overview]. 

Affirmation:  An  oral  or  written  statement  confirming  or  supporting  implementation  of  a  CMMI 
model  practice.  Affirmations  are  usually  provided  by  the  implemented  of  the  practice  and/or 
internal  or  external  customers,  but  may  also  include  other  stakeholders  (e.g.,  managers,  suppliers) 
[derived  from  MDD  method  overview].  Interview  responses  are  examples  of  oral  affirmations. 
Alternative  forms  of  affirmations  could  include  presentations  or  demonstrations  of  a  tool  or 
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mechanism  as  it  relates  to  implementation  of  a  CMMI  model  practice  (e.g.,  instruments  including 
questionnaires,  surveys,  PII  descriptions,  and  other  written  information  that  indicates  practice 
implementation)  [derived  from  MDD  PII  appendix  B], 

These  data  definitions  are  a  hallmark  of  the  SCAMPI  family  of  appraisal  methods.  Each  method 
can  be  made  to  work  more  efficiently  by  utilizing  an  organized  structure  for  storing  and 
presenting  the  objective  evidence  needed  for  the  appraisal.  Practice  Implementation  Indicators 
(PIIs)  are  one  way  objective  evidence  can  be  identified  and  documented  in  advance  and  in 
accordance  with  the  definitions  above.  When  a  sufficient  amount  of  such  pre-defmed  evidence  is 
available,  the  appraisal  may  be  conducted  in  what  is  termed  “verification  mode.”  In  verification 
mode  an  appraisal  team  does  not  need  to  search  for  data,  only  to  verify  the  data  presented. 
However,  when  such  evidence  is  not  prepared  in  advance,  the  appraisal  is  typically  carried  out  in 
what  is  termed  “discovery  mode,”  because  those  conducting  the  appraisal  must  search  for  and 
discover  the  information  and  artifacts  that  can  serve  as  objective  evidence. 

There  are  three  general  uses  for  the  concept  and  structure  of  PIIs: 

•  as  a  suggested  structure  and  content  guide  for  “evidence”  provided  as  input  to  the  appraisal. 
The  PII  structure  maps  the  objective  evidence  to  the  model  and  organization,  facilitating  the 
work  of  the  appraisal  team 

•  as  a  framework  for  data  collection  planning,  progress  tracking,  and  data  interpretation 

•  as  a  format  for  summarizing  appraisal  data  and  results  for  future  use 

One  strategy  for  using  SC  AMPI  methods  is  to  build  a  set  of  Practice  Implementation  Indicators 
gradually,  through  a  series  of  informal  appraisals.  A  baseline  appraisal  can  be  conducted  at 
different  points  to  ensure  that  an  efficient  verification-based  appraisal  is  supported  by  the 
organization’s  knowledge  of  its  own  practices. 

Types  of  Objective  Evidence:  Interviews  and  Documents 

The  combination  of  desired  depth  of  model  coverage  and  the  breadth  of  organizational  coverage 
sought  will  mandate  the  use  of  different  types  of  objective  evidence  (e.g.,  documents  or 
interviews).  Interviews  may  take  a  collaborative  workshop  approach,  with  relaxed  coverage 
requirements  for  objective  evidence.  Interviews  can  provide  a  very  comprehensive  source  of  data 
in  an  appraisal,  as  they  permit  interviewers  to  change  focus  “on  the  fly”  to  pursue  practices 
described  by  the  interviewees.  Interviews  may  be  selected  as  the  primary  data  type  in  many 
implementations  of  SCAMPI  B  and  C.  Detailed  review  of  artifacts  can  provide  evidence  that 
represent  objectively  verifiable  data.  Documents,  which  are  often  in  electronic  form,  can  help 
achieve  greater  coverage  when  paired  with  well  planned  interviews.  It  is  this  pattern  of  multiple 
converging  data  types  and  the  sufficiency  of  the  sample  used  (coverage)  that  support  the  strength 
of  judgments  made  at  the  conclusion  of  the  appraisal. 

By  tailoring  the  depth  of  coverage  for  both  the  organization  and  model,  the  SCAMPI  C  method 
can  be  configured  as  a  bare-bones  appraisal  carried  out  by  a  single  expert  consultant,  interviewing 
people  over  the  course  of  a  day,  and  yielding  findings  of  strengths  and  weaknesses.  The  SCAMPI 
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C  method  can  also  be  configured  as  a  week-long  event  in  which  detailed  reviews  of  documents 
are  combined  with  interviews  of  a  substantial  cross-section  of  the  staff  in  the  organization.  The 
choice  to  perform  this  latter,  more  robust,  appraisal  as  a  SCAMPI  C  may  be  prompted  by  the 
intent  to  use  a  single  expert  appraiser,  or  to  use  a  collaborative  approach  in  which  the 
interviewees  are  enlisted  to  brainstorm  new  approaches,  rather  than  providing  objective  evidence. 

The  SCAMPI  B  method  carries  a  requirement  for  interview  data  as  well  as  direct  artifacts.  There 
is  a  higher  standard  of  data  coverage  to  meet  in  the  minimal  configuration  of  the  SCAMPI  B 
method  compared  to  the  minimal  configuration  of  the  SC  AMPI  C  method.  The  SCAMPI  B 
method  requires  the  use  of  a  team  and  a  consensus-driven  process.  The  desire  to  avoid  ratings  is 
sometimes  the  key  reason  to  select  SCAMPI  B.  In  such  cases,  the  data  coverage  may  meet  (or 
exceed)  the  standards  defined  for  SCAMPI  A. 

In  summary,  the  use  of  a  single  data  type,  or  a  single  appraiser,  in  order  to  minimize  the  cost  of  an 
appraisal  will  necessarily  lead  to  the  SCAMPI  C  method.  When  more  than  one  data  type  is  used, 
one  of  them  being  interviews,  and  a  consensus-based  appraisal  is  desired,  the  SCAMPI  B  is  more 
frequently  used. 

Practice  Characterization 

CMMI  establishes  goals  as  required  components,  and  the  practices  that  relate  to  each  goal  are 
treated  as  expected  components.  It  is  important  to  view  the  practices  in  the  model  as  examples. 
When  characterizing  practices,  the  intent  is  to  describe  the  extent  to  which  the  organization  has 
accounted  for  how  each  practice  contributes  to  the  achievement  of  the  goal  to  which  it  relates. 

The  nature  of  this  contribution  is  governed  by  the  context  in  which  the  process  must  be  defined  or 
implemented.  It  is  tempting  to  try  to  find  a  one-for-one  mapping  between  model  practices  and 
practices  implemented  in  the  organization.  This  is  often  unrealistic.  Organizations  frequently  have 
multiple  practices,  sometimes  from  different  parts  of  the  organization,  that  when  taken  as  a  whole 
address  the  intent  of  the  CMMI  practice  and  contribute  to  goal  satisfaction.  A  one-for-one 
mapping  is  not  a  necessary  pre-condition  to  characterization  of  model  practices.  The 
organization’s  objective  evidence,  and  other  appraisal  data,  are  usually  mapped  to  the  model 
structure,  rather  than  prescribing  the  implementation  of  practices  and  functions  as  specified  in  the 
model. 

While  practice  characterizations  apply  only  to  practices,  these  outcomes  can  be  grouped  by  model 
components,  such  as  process  areas  or  organizational  groups  such  as  projects.  Each  SCAMPI  class 
has  a  characterization  scale  that  is  designed  to  match  the  minimum  standards  of  data  sufficiency 
for  that  method.  These  scales  are  defined  in  the  following  table. 
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Optional  “Fidelity”  Scale  for  SCAMPI  C 

Required  Scale  for  SCAMPI  B 

Low 

The  intent  of  the  model  practice  is 
judged  absent,  or  inadequately 
addressed  in  the  approach;  goal 
achievement  is  judged  unlikely 
because  of  this  absence  or 
inadequacy. 

Red 

The  intent  of  the  model  practice  is 
judged  to  be  absent  or  poorly 
addressed  in  the  set  of  implemented 
practices;  gaps  or  issues  that  will 
prevent  goal  achievement,  if  the 
deployment  occurred  in  this  way 
across  the  organizational  unit,  were 
identified. 

Medium 

The  intent  of  the  model  practice  is 
judged  to  be  partially  addressed 
in  the  approach,  and  only  limited 
support  for  goal  achievement  is 
evident. 

Yellow 

The  intent  of  the  model  practice  is 
judged  to  be  partially  addressed  in  the 
set  of  implemented  practices;  some 
gaps  or  issues  were  identified,  which 
might  threaten  goal  achievement  if  the 
deployment  occurred  in  this  way 
across  the  organizational  unit. 

High 

The  intent  of  the  model  practice  is 
judged  to  be  adequately  addressed 
in  the  set  of  practices  (planned  or 
deployed),  in  a  manner  that 
supports  achievement  of  the  goal 
in  the  given  process  context. 

Green 

The  intent  of  the  model  practice  is 
judged  to  be  adequately  addressed  in 
the  implemented  set  of  practices 
examined,  in  a  manner  that  would 
support  goal  achievement,  if  the 
practice  were  deployed  across  the 
organizational  unit. 

In  addition  to  the  above,  a  designation  of  “out  of  scope”  is  used  when  no  characterization  was 
assigned  because  the  appraisal  did  not  gather  data  to  support  characterization  of  the  practice. 

The  purpose  of  the  characterization  scale  is  to  support  comparisons  of  different  model 
components,  or  of  different  elements  of  the  organizational  unit.  These  differences  draw  attention 
to  things  that  represent  significant  improvement  opportunities.  The  absolute  value  of  any  given 
characterization  result  is  of  limited  value.  The  characterization,  along  with  a  link  to  the  evidence 
reviewed  and  the  strengths  and  weaknesses  found,  can  be  very  meaningful.  The  contrast  among 
the  characterization  values  conveys  the  beneficial  information  provided  by  this  technique. 

Findings  and  Other  Appraisal  Results 

The  vast  majority  of  appraisals  performed  using  the  CMM  and  CMMI  result  in  a  slide 
presentation  that  catalogs  or  summarizes  statements  of  strengths,  weaknesses,  or  other  descriptive 
findings.  This  presentation  is  typically  structured  in  accordance  with  the  structure  of  the  model, 
and  its  length  is  determined  by  the  model  scope  of  the  appraisal.  The  SCAMPI  process  requires 
analysis  at  a  very  detailed  level.  The  high-level  results  typically  found  in  the  findings  presentation 
are  summaries  based  on  individual  comparisons  between  practices  in  CMMI  and  the  planned  or 
implemented  practices  in  the  organizational  unit. 
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Beyond  the  findings  presentation,  SCAMPI  B  and  C  appraisals  can  provide  profiles  of  practice 
characterization  results  summarized  in  categories  determined  by  elements  of  the  model  or 
elements  of  the  organizational  unit  (or  both).  A  spreadsheet  containing  the  red/yellow/green 
characterizations  for  elements  (  model  and/or  organizational)  within  the  scope  of  the  appraisal  is 
often  integrated  with  the  findings  presentation.  The  patterns  in  the  characterization  data  are  used 
to  add  emphasis  or  elaborate  the  findings  statements  as  appropriate.  This  detailed,  project  by 
project,  practice  by  practice,  data  collection  and  analysis  has  proven  to  be  a  very  powerful 
approach.  The  data  and  judgments  supporting  the  high-level  findings  can  be  recorded  for  later 
review  and  use. 

Findings  mapped  to  individual  elements  of  the  model,  as  well  as  individual  elements  of  the 
organization,  may  be  reported.  Findings  that  do  not  specifically  relate  to  the  model  but  represent 
significant  issues  or  strengths  in  the  organization  may  also  be  reported.  Findings  that  focus  on  the 
contrast  between  the  organizational  standards  and  project  implementations  may  be  selected  as 
well.  Finally,  it  is  permissible  to  define  custom  specifications  for  appraisal  outputs  such  as 
“anticipated  return  on  investment  from  implementing  the  practice”  or  “estimated  cost  of 
implementing  the  practice.” 

A  simple  matrix  or  database  of  Practice  Implementation  Indicators  may  be  developed 
incrementally  over  the  conduct  of  several  SCAMPI  C  or  SCAMPI  B  appraisals.  The  less  formal 
environment  of  the  class  C  can  serve  as  an  opportunity  to  brainstorm  with  members  of  the 
organization.  In  a  class  B  appraisal,  the  identified  direct  and  indirect  artifacts  can  be  reviewed  for 
relevance  and  appropriateness.  Over  time,  the  organization’s  confidence  in  the  quality  of  the  set 
of  Practice  Implementation  Indicators  improves,  as  more  appraisals  are  performed  and  the 
database  is  incrementally  refined. 

Method  Flow 

At  the  most  general  level,  every  SCAMPI  appraisal  has  three  primary  phases:  (1)  plan  and 
prepare  for  the  appraisal,  (2)  conduct  the  appraisal,  and  (3)  report  the  results.  Within  this 
structure,  the  feasibility  of  the  plan  for  the  appraisal  event  must  be  evaluated  during  planning  and 
throughout  the  conduct  of  the  appraisal.  The  focus  of  data  collection  activities  may  require 
adjustments  based  on  results  from  tracking  against  the  defined  set  of  desired  information. 
Analysis  of  objective  evidence,  as  well  as  other  data  collected  for  the  appraisal,  transforms  the 
data  into  appraisal  results  that  are  then  documented  and  reported  to  appropriate  stakeholders. 

The  Appraisal  Method  User  Perspective 

Guidance  for  tailoring  SCAMPI  B  and  C  methods  is  based  on  the  perspective  of  the  appraisal 
method  user,  who  can  be  an  internal  change  agent,  a  professional  consultant,  or  an  external 
auditor.  This  perspective  has  a  strong  influence  on  the  tailoring  decisions.  The  differences  in 
tailoring  occur  most  significantly  in  the  planning  processes  (i.e.,  establishing  the  appraisal 
objectives,  sponsorship,  appraisal  planning,  and  selection  of  participants)  and  reporting  processes 
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(i.e.,  reporting  appraisal  results,  use  of  results  for  decision  making,  and  follow-on  activities).  Sets 
of  tailoring  options  can  be  bundled  together  for  application  in  specific  domains. 

Internal  Change  Agent 

The  internal  change  agent  often  works  within  the  organizational  unit  being  appraised.  The 
internal  change  agent  might  be  a  member  of  the  appraised  organization  or  an  employee  of  the 
company  but  external  to  the  appraised  organization.  Appraisals  in  this  context  are  commonly  used 
to  monitor  implementation  of  process  improvement  actions,  determine  readiness  for  a  more 
formal  appraisal  event,  and  to  confirm  remediation  of  weaknesses  identified  in  previous  appraisal 
activities.  Knowledge  of  the  organization  -its  structure,  the  dissemination  of  roles  and 
responsibilities,  and  the  history  of  individual  people-  provides  a  basis  for  insightful  scoping  and 
sampling  decisions.  The  internal  change  agent  knows  the  lay  of  the  land  and  where  the 
“skeletons”  are. 

Professional  Consultant 

The  professional  consultant  is  typically  brought  in  on  a  temporary  basis,  though  most  consulting 
firms  have  ongoing  business  relationships  with  their  clients.  Professional  consultants  are  typically 
used  to  enhance  the  perception  of  objectivity  of  appraisal  results  through  use  of  a  third  party,  to 
integrate  highly  specialized  knowledge  and  skills,  or  to  transfer  competencies  to  internal  staff  by 
their  participation  in  a  team  led  by  an  expert.  The  findings  and  results  provided  are  often  used  to 
initiate  a  program  of  model-based  improvement  or  to  monitor  progress  at  meaningful  milestones. 

External  Auditor 

The  external  auditor  may  be  from  the  same  company  as  the  appraised  organization,  but 
performing  an  auditing  function;  from  a  company  considering  or  monitoring  the  appraised 
organization  as  a  supplier;  or  from  a  government  organization  engaged  in  source  selection  or 
contract  monitoring  activities.  While  each  of  these  appraisal  users  has  an  auditing  responsibility, 
the  government  auditor  has  perhaps  the  most  stringent  environment  in  which  to  work.  For  this 
reason,  much  of  the  external  auditor  guidance  offered  in  this  handbook  pertains  to  the  often 
unique  requirements  placed  on  a  government  auditor.  The  government  auditor,  for  example,  uses 
appraisals  to  examine  organizations’  processes  as  input  to  a  decision  regarding  future  business. 
The  two  primary  uses  of  these  appraisals  are  for  source  selection  and  contract  process  monitoring. 
In  source  selection,  appraisal  results  are  used  as  a  high-value  discriminator  to  select  suppliers. 

The  results  are  used  in  characterizing  the  process-related  risk  of  awarding  a  contract  to  a  supplier. 
The  appraisal  results  are  typically  one  criterion  among  many  used  to  select  a  supplier.  Results  are 
often  used  to  establish  a  baseline  that  is  used  in  subsequent  process  monitoring  with  the  selected 
supplier.  In  contract  process  monitoring,  appraisal  results  are  used  as  input  for  an  incentive/award 
fee  decision  or  to  monitor  implementation  of  a  risk  management  plan.  The  appraisal  results  are 
used  to  help  the  sponsoring  organization  tailor  contract  or  process  monitoring  efforts  by  focusing 
on  the  observed  strengths  and  weaknesses  of  the  supplier  organization’s  processes.  These 
activities  may  be  useful  for  commercial  organizations  in  their  selection  of  suppliers. 
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Chapter  2: 

Process  Descriptions 

Appraisal  Processes 

ARC.4.1.2 

ARC.4.1.2.a 

ARC.4.2.10 

The  SCAMPI  B  and  C  appraisal  methods  are  defined  according  to 
three  major  phases  that  comprise  any  given  appraisal.  The  phases  are 
(1)  plan  and  prepare  for  appraisal,  (2)  conduct  appraisal,  and  (3)  report 
results.  These  phases  each  contain  a  set  of  processes  that  can  be 
tailored,  within  certain  parameters,  by  the  user. 

This  chapter  provides  definitions  of  the  required  practices,  parameters 
and  limits  of  tailoring,  and  guidance  associated  with  each  of  the 
processes.  The  responsibilities  of  the  appraisal  (team)  leader  are  also 
detailed  in  this  chapter. 

When  the  required  practices  are  performed  in  accordance  with  the 
specified  parameters  and  limits,  valid  enactments  of  SCAMPI  B  or 
SCAMPI  C  are  assured. 

1.  Plan  and  Prepare  for 
Appraisal 

ARC4.3 

This  phase  consists  of  the  following  processes 

1 . 1  Analyze  Requirements 

1.2  Develop  Appraisal  Plan 

1.3  Select  and  Prepare  Team 

1.4  Prepare  Participants  &  Obtain  Initial  Objective  Evidence 

1.5  Prepare  for  Collection  of  Objective  Evidence 

2.  Conduct  Appraisal 

This  phase  consists  of  the  following  processes 

2.1  Examine  Objective  Evidence 

2.2  Document  Objective  Evidence 

2.3  Verify  Objective  Evidence 

2.4  Validate  Preliminary  Appraisal  Outputs 

2.5  Generate  Appraisal  Results 

3.  Report  Results 

ARC4.7 

This  phase  consists  of  the  following  processes 

3.1  Deliver  Appraisal  Results 

3.2  Package  and  Archive  Appraisal  Assets 

Summary 

Each  of  the  3  phases  and  12  processes  outlined  above  is  covered  in  the 
remaining  sections  of  this  chapter. 
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Phase  1 

Plan  and  Prepare  for  Appraisal 

Overview 

The  minimum  requirements  for  conducting  planning  processes  and 
creating  artifacts  are  specified  in  this  section.  This  phase  consists  of  the 
following  processes: 

1.1  Analyze 
Requirements 

The  appraisal  input  provides  the  foundation  for  planning  and 
conducting  the  appraisal  activities  in  a  way  that  maximizes 
achievement  of  the  appraisal  sponsor’s  goals.  The  appraisal  input  must 
be  documented  and  baselined  (with  the  sponsor’s  signature)  prior  to 
the  start  of  data  collection  that  targets  information  related  to  model 
practices. 

“Sponsor”  may  be  a  different  type  of  person  when  we  consider 
different  usage  modes.  In  source  selection,  it  is  someone  in  the  source 
selection  organization  rather  than  a  senior  manager  at  the  site(s)  where 
appraisal  activities  are  carried  out. 

1.2  Develop  Appraisal 
Plan 

ARC4.3.5 

The  appraisal  plan  forms  the  basis  for  communicating  and  maintaining 
commitments.  Review  by  stakeholders  and  sponsor  signoff  are 
required. 

1.3  Select  and  Prepare 
Team 

For  the  SCAMPI  B  method,  minimum  standards  for  team  composition 
and  team  preparation  are  described  in  detail.  The  SCAMPI  C  method 
does  not  require  that  a  team  be  used. 

1.4  Prepare 

Participants  and 

Obtain  Initial  Objective 
Evidence 

Preparing  appraisal  participants  to  contribute  effectively  to  the 
appraisal  requires  consideration  of  the  roles  they  are  to  play.  The 
appraisal  plan  must  include  activities  intended  to  communicate  such 
information  to  the  appraisal  participants. 

1.5  Prepare  for 
Collection  of  Objective 
Evidence 

The  conduct  of  a  readiness  review  is  a  requirement  for  all  SCAMPI 
appraisals.  The  focus  and  conduct  of  the  readiness  review  will  vary 
depending  on  which  method  is  being  used. 
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1.1  Analyze  Requirements 


Overview  The  needs  and  constraints  of  the  appraisal  sponsor  must  be  determined 

in  order  to  best  tailor  the  appraisal  method.  The  initial  set  of 
parameters  that  guide  appraisal  planning  must  be  collected  in  an 
appraisal  input  document.  When  the  appraisal  planning  is  complete,  all 
of  the  information  from  the  appraisal  input  may  migrate  to  the 
appraisal  plan.  In  fact,  use  of  a  “draft  appraisal  plan”  as  the  vehicle  for 
documenting  and  obtaining  commitment  to  the  appraisal  input  is 
permitted,  though  not  recommended  unless  the  appraisal  is  to  be  a  very 
short  event  that  involves  only  a  few  people  (e.g.,  fewer  than  10 
interviewees). 


The  appraisal  team  leader  shall 

•  Meet  or  correspond  with  the  appraisal  sponsor 

•  Seek,  clarify,  and  verify  appraisal  requirements 

•  Document  and  maintain  the  appraisal  input 

•  Obtain  sponsor  approval  of  the  appraisal  input 


Required  Practices 

ARCAl.l.e 


Parameters  and  Limits: 
General 

ARC.4.2.2.b 

ARC.4.3.2 

ARC.4.3.4 


At  least  one  interaction  between  the  sponsor  and  the  team  leader — 
whether  that  interaction  occurs  in  person,  via  telephone,  or  through 
written  correspondence — is  required  in  advance  of  final  approval  of  the 
appraisal  input.  The  suitability  of  the  method  selected  (  SCAMPI  B  or 
SCAMPI  C)  must  be  discussed. 


Each  of  the  elements  of  the  appraisal  input  (detailed  below)  must  be 
documented  and  approved  by  the  appraisal  sponsor.  The  method  for 
packaging  this  information  is  to  be  selected  by  the  appraisal  (team) 
leader — however,  use  of  the  appraisal  input  template,  provided  by  the 
SEI,  is  strongly  recommended. 

Final  approval  of  the  appraisal  input  must  be  designated  by  the 
sponsor,  who  must  sign  the  document  that  contains  the  information. 
The  team  leader  also  signs  the  appraisal  input  document  to  indicate 
his/her  understanding  and  commitment.  This  sign-off  must  occur 
before  any  interviews  or  artifact  reviews  may  begin.  Electronic 
signatures  (e.g.,  an  email  indicating  commitment  to  the  appraisal  input) 
are  acceptable  as  long  as  they  clearly  identify  the  document  upon 
which  the  agreement  is  based — typically  by  naming  the  document  and 
its  version  number  or  publication  date.  The  document  containing  the 
electronic  signature  must  be  printed  and  maintained  with  the  appraisal 
artifacts  as  an  element  of  the  appraisal  input. 


Con  tin  ued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and 
Limits:  Required 
Contents  of  Appraisal 
Input 

ARC  4.1.1.a 
ARC.4.2.2.a 
ARC4.3.3.a-n 
ARC4.3.4 


The  appraisal  (team)  leader  and  the  sponsor  agree  and  coordinate 
revisions  to  the  documented  appraisal  input,  which  must  include  every 
item  listed  below  (Note:  items  marked  with  an  asterisk  [*]  are 
elaborated  in  subsequent  Parameters  and  Limits  sections): 

a.  the  identity  of  the  sponsor  of  the  appraisal,  and  the  sponsor’s 
relationship  to  the  organizational  unit  being  appraised 

b.  the  appraisal  purpose,  including  alignment  with  business 
objectives 

c.  the  appraisal  reference  model  scope* 

d.  the  organizational  unit  that  is  the  subject  of  the  appraisal* 

e.  the  process  context* 

f.  the  appraisal  constraints* 

g.  the  CMMI  models  used,  including  the  version,  discipline,  and 
representation  (staged  or  continuous) 

h.  a  written  affirmation  that  the  appraisal  (team)  leader  meets  the 
minimum  criteria  specified  by  the  SEI  for  leading  SCAMPI  B 
and  C  appraisals* 

i.  the  identity  and  affiliation  of  the  appraisal  team  members, 
including  the  appraisal  (team)  leader,  with  their  specific 
appraisal  responsibilities  (if  a  team  is  used)* 

j.  the  identity  (name  and  organizational  affiliation)  of  appraisal 
participants  and  support  staff,  with  their  specific 
responsibilities  for  the  appraisal.  For  early  drafts  of  the 
appraisal  input,  these  participants  may  be  specified  by  role, 
rather  than  by  name.  However,  the  names  of  all  participants 
must  be  documented  for  sponsor  review  and  approval  prior  to 
the  start  of  interviews. 

k.  any  additional  information  to  be  collected  during  the  appraisal 
to  support  achievement  of  the  appraisal  objectives 

l.  a  description  of  the  planned  appraisal  outputs,  including 
outputs  to  be  validated,  and  stakeholders  who  will  participate  in 
the  validation  and  those  who  will  ultimately  receive  individual 
appraisal  outputs 

m.  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action 
plans,  re-appraisal)* 

n.  planned  tailoring  of  the  appraisal  method  and  associated 
tradeoffs,  including  the  sample  size  or  coverage  of  the 
organizational  unit  (Please  refer  to  the  Guidance  section  on 
tailoring,  below,  as  well  as  the  contents  of  Chapter  2  of  this 
document  for  detailed  information  about  tailoring.) 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and  Limits: 
Model  Scope 

ARC.4.2.1.a 

ARC.4.2.1.b 

ARC.4.2.3.a-b 

ARC.4.3.3.C 


Parameters  and  Limits: 
Organizational  Unit 

ARC.4.1.1.b 

ARC4.2.4.a-d 


Parameters  and  Limits: 
Process  Context 

ARC4.3.3.e 


The  SCAMPI  B  and  SCAMPI  C  methods  are  based  on  the  Appraisal 
Requirements  for  CMMI  (ARC)  Version  1.1  and  are  intended  for  use 
with  all  variants  of  models  published  with  Version  1.1  of  the  CMMI 
product  suite. 

The  appraisal  input  must  specify  the  detailed  scope  of  the  model  under 
consideration  in  the  appraisal.  It  is  expected  that  a  list  of  process  areas 
and  the  capability  level  scope  (when  the  continuous  model 
representation  is  used)  will  be  documented.  However,  if  portions  of 
process  areas  are  sampled  (to  the  exclusion  of  the  rest  of  the  process 
area),  then  a  more  detailed  specification  of  the  exact  model  scope  must 
be  documented — especially  if  some  model  components  will  be  covered 
with  data  relating  to  only  a  subset  of  the  organizational  unit. 


A  description  of  the  organizational  unit  must  be  documented.  The 
written  description  must  include  identification  of 

•  projects  selected  for  inclusion 

•  functional  elements  selected  for  inclusion 

•  names  of  individuals  sampled  for  participation  and  their  affiliation 
to  the  included  projects  or  functional  elements 


The  documentation  of  process  context  must  include 

1 .  the  size  of  the  organizational  unit 

2.  the  demographics  of  the  organizational  unit 

3.  the  application  domain  of  the  products  or  services  of  the 
organizational  unit 

4.  the  size,  criticality,  and  complexity  of  products  or  services 

These  are  attributes  of  the  organization  that  help  form  the  context  in 
which  engineering  and  management  processes  must  be  deployed  and 
maintained.  These  contextual  attributes  therefore  must  inform  the 
interpretation  of  implemented  practices  as  they  are  compared  to 
CMMI. 

In  the  use  of  SCAMPI  B  or  SCAMPI  C  in  an  acquisition  or  process 
monitoring  context,  there  may  be  a  need  for  more  contextual 
information.  In  such  settings,  there  is  often  a  broader  context  in  which 
the  appraised  organization  must  operate,  and  this  context  can  have  a 
bearing  on  the  appraisal  planning.  For  example,  the  rationale  for 
selecting  particular  projects  or  participants  might  be  tied  to  their  future 
involvement  in  a  contract  under  competition.  Similarly,  the 
involvement  of  a  part  of  the  organization  with  a  corporate  initiative 
may  account  for  differences  observed  among  participating  groups. 
Items  of  this  type  must  be  documented  in  the  appraisal  input. 


Con  tin  ued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and  Limits: 
Appraisal  Constraints 

ARC.4.1.1.d 

ARC.4.3.3.f 


The  documented  appraisal  constraints  must  address 

1.  availability  of  key  resources 

2.  schedule  constraints 

3.  the  maximum  amount  of  time  to  be  used  for  the  appraisal 

4.  specific  process  areas  or  organizational  entities  to  be  excluded 
from  the  appraisal  if  the  documented  model  scope  of  the 
appraisal  does  not  already  make  this  clear 

5.  the  minimum,  maximum,  or  specific  sample  size  or  coverage 
that  is  desired  for  the  appraisal  if  there  are  such  constraints 
communicated  or  implied  by  the  sponsor 

6.  the  ownership  of  the  appraisal  outputs  and  any  restrictions  on 
their  use 

7.  controls  on  information  resulting  from  a  confidentiality 
agreement 

8.  non-attribution  of  appraisal  data  to  associated  sources  (see  the 
next  section  on  Confidentiality  and  Non- Attribution) 

The  first  five  constraints  typically  form  the  initial  input  to  planning  and 
represent  areas  where  compromise  may  be  negotiated  to  meet  the 
objectives  of  the  appraisal  sponsor.  The  last  three  constraints  reflect 
efforts  to  protect  the  integrity  and  credibility  associated  with  the 
appraisal  process.  These  are  elaborated  more  fully  in  the  Parameters 
and  Limits  section  below. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and  Limits: 
Confidentiality  and 
Non-Attribution  of  Data 
Sources 

ARC.4.1.1.C 

ARC4.2.15 


Confidentiality  of  appraisal  data  and  appraisal  results  is  a  principle  that 
serves  to  enhance  the  integrity  and  credibility  of  the  process.  This 
principle  mandates  that  the  appraisal  sponsor  is  the  owner  of  the 
appraisal  outputs.  It  also  mandates  that  intermediate  work  products  of 
the  appraisal  are  to  be  viewed  only  by  team  members,  and  that  specific 
information  (raw  data)  provided  by  individuals  is  to  be  kept 
confidential,  except  as  it  appears  in  the  appraisal  results.  The  principle 
of  confidentiality  should  be  distinguished  from  the  principle  of  non¬ 
attribution. 

In  no  case  shall  the  appraisal  be  designed  in  such  a  way  as  to  identify, 
in  the  appraisal  results,  individual  participants  with  the  specific  data 
they  provide.  This  non-attribution  requirement  extends  to  a  prohibition 
from  associating  (explicitly)  the  names  of  people  providing  data  and 
the  characterizations  or  findings  that  derive  from  those  data.  This  does 
not  preclude  the  potential  for  identifying  groups  or  projects  to  which 
appraisal  findings  apply,  as  long  as  the  individuals  who  provide  data 
leading  to  those  findings  are  not  identified  with  the  data  they  provide. 

In  some  cases,  the  sponsor  may  request  project-specific  results,  or 
results  organized  by  some  other  grouping  of  organizational 
participants.  Such  lower-level  aggregations  of  data  are  supported  by 
the  SCAMPI  B  and  SCAMPI  C  methods.  However,  the  intent  to 
summarize  the  data  in  this  way  must  be  made  clear  in  advance  to  all 
appraisal  participants. 

All  appropriate  mitigation  steps  must  be  taken  to  minimize  the 
possibility  of  attributing  data  or  results  to  individuals  within  the 
organization.  This  may  include  subtle  considerations  in  the  way 
findings  are  worded,  or  aggregating  the  results  of  several  small  groups 
that  would  stand  out  among  others.  In  no  case  shall  the  agreed-upon 
reporting  strategy  be  changed  during  the  course  of  the  appraisal  if  the 
potentially  affected  individuals  have  already  provided  data  under  an 
existing  agreement  for  its  use. 

A  more  stringent  set  of  confidentiality  provisions  is  likely  to  be 
required  for  use  of  these  appraisal  methods  in  an  acquisition  context. 


Con  tin  ued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and  Limits: 
Identification  of  the 
Appraisal  Team  and 
Appraisal  Team  Leader 


Parameters  and  Limits: 
Appraisal  Input 
Template 


Parameters  and  Limits: 
Anticipated  Follow-on 
Activities 


The  identification  of  the  appraisal  team  and  the  appraisal  team  leader 
must  include  (by  explicit  statement  or  by  reference)  the  qualifications 
they  have  met  in  order  to  serve  in  their  designated  roles.  These 
qualifications  include  prerequisite  engineering  and  management 
experience  as  well  as  the  successful  completion  of  prerequisite 
training.  Please  refer  to  the  information  under  Process  1.3,  Select  and 
Prepare  Team,  for  more  elaboration  of  this  topic. 

In  an  acquisition  context,  prerequisites  pertaining  to  applicable 
regulations  or  provisions  for  security  clearances  may  also  be  relevant. 
It  is  required  that  all  such  matters  be  documented  in  the  appraisal  input, 
even  if  some  information  of  this  type  would  have  to  be  omitted  before 
the  document  could  be  provided  to  others  (e.g.,  submitted  to  the  SEI). 

The  SCAMPI  C  method  does  not  require  use  of  an  appraisal  team.  In 
the  appraisal  input  for  this  class,  only  the  appraisal  leader  must  be 
identified. 


An  appraisal  input  template  is  available  in  the  materials  provided  to 
SCAMPI  Lead  AppraisersSM.  Use  of  this  template  is  strongly 
recommended.  One  might  choose  not  to  use  this  template  if  there  is  a 
different  template  already  in  routine  use  within  the  organization  for 
communicating  this  type  of  detailed  information.  In  addition,  one 
might  choose  to  use  a  preliminary  draft  of  an  appraisal  plan  (based  on 
an  existing  template)  to  document  and  communicate  this  information 
rather  than  creating  an  entirely  separate  document  for  the  appraisal 
input. 


The  appraisal  input  document  must  specify  the  follow-on  activities  that 
have  a  bearing  on  the  conduct  of  the  appraisal.  Examples  of  such 
activities  include  the  generation  of  a  report  summarizing  the  appraisal 
outcome  or  providing  recommendations  for  improvement.  In  addition, 
the  linkage  of  the  appraisal  to  future  appraisals  to  be  conducted  in  the 
same  organization  should  be  documented  in  the  appraisal  input. 
Activities  not  directly  tied  to  the  appraisal  event,  such  as  process 
improvement  plans,  strategies  at  a  corporate  level,  or  acquisition- 
related  events,  are  not  expected  to  be  documented  in  the  appraisal 
input. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Parameters  and  Limits: 
Tailoring  the  Appraisal 
Method 


Guidance:  Model 
Scope 


The  SCAMPI  family  of  appraisal  methods  provides  for  a  great  deal  of 
flexibility  in  customizing  the  application  of  the  method  to  meet  the 
needs  of  sponsors  and  organizations.  The  appraisal  input  must 
document  important  decisions  about  tailoring  the  method,  which  bear 
on  how  the  results  must  be  interpreted.  All  decisions  regarding 
scoping,  in  terms  of  the  model  and  the  organizational  unit,  must  be 
clearly  documented.  In  addition,  all  decisions  regarding  the  structure 
and  content  of  the  output  must  be  documented,  because  understanding 
this  information  may  affect  the  way  some  appraisal  participants  will 
respond  to  requests  for  information. 

Significant  decisions  regarding  the  nature  of  data  sought  by  the  team, 
and  the  standards  by  which  these  data  are  treated,  must  also  be 
documented  in  the  appraisal  input.  This  is  not  to  say  that  the  specific 
database  structure  used  must  be  communicated  in  the  appraisal  input. 
However,  if  participants  are  expected  to  provide  actual  project  artifacts 
(live  data)  and  not  just  “process  documentation,”  then  this  must  be 
understood  during  planning.  Similarly,  if  the  team  expects  to  update  a 
set  of  practice  implementation  indicator  descriptions  (supplied  by  the 
organizational  unit  at  the  start  of  the  appraisal),  then  this  fact  must  be 
documented  in  the  appraisal  input. 

Obviously,  many  detailed  choices  (e.g.,  where  to  eat  lunch,  specific 
questions  to  ask  during  interviews)  will  not  be  documented  in  the 
appraisal  input.  However,  it  is  expected  that  a  thorough  review  of  the 
Required  Practices  and  Parameters  and  Limits  sections  of  this 
document  will  be  performed  during  planning  in  order  to  identify 
tailoring  decisions  that  warrant  inclusion  in  the  appraisal  input. 


CMMI-based  appraisals  typically  specify  the  process  areas  to  be 
considered  during  the  appraisal,  along  with  the  highest  expected  rating. 
With  SCAMPI  B  and  SCAMPI  C,  it  is  permissible  to  sample  model 
content  at  a  more  detailed  level.  Because  no  goal  ratings  or 
capability/maturity  level  ratings  are  permitted  in  classes  B  and  C,  there 
is  no  requirement  to  include  an  entire  process  area,  although  it  is 
expected  that  most  appraisals  will  sample  according  to  process  areas. 
For  example,  a  sponsor  might  choose  to  examine  a  single  goal  (and  its 
associated  practices)  within  a  particular  process  area,  or  a  set  of 
generic  practices  associated  with  one  or  more  process  areas. 
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1.1  Analyze  Requirements  (continued) 


Guidance: 
Organizational  Unit 


Guidance:  Appraisal 
Input 


Objectively  documenting  the  definition  of  the  organizational  unit 
clarifies  the  portion  of  the  organization  to  which  the  appraisal  results 
can  be  legitimately  generalized.  The  extent  to  which  the  sampled 
appraisal  participants  adequately  represent  the  views  of  the 
organizational  unit,  as  a  whole,  may  be  estimated  during  data 
collection  in  order  to  gauge  generalizability  of  appraisal  outputs. 

Some  appraisals  are  applied  to  organizational  units  defined  narrowly  as 
a  set  of  four  projects.  Other  appraisals  sample  practitioners  and 
managers  from  across  the  organizational  unit,  as  well  as  the  people 
within  the  individual  projects  identified  to  comprise  the  organizational 
unit.  This  type  of  sampling  strategy  may  tend  to  improve  the  veracity 
of  the  appraisal  outcomes.  The  sampling  strategy  must  be  documented 
in  the  appraisal  input. 

The  initial  draft  of  the  appraisal  input  may  lack  some  details  like  the 
names  of  individuals  sampled  for  participation.  However,  a  list  of 
projects  and  functional  elements  selected — or  the  set  from  which  the 
final  selection  will  be  made — is  typically  specified.  Not  every  project 
involved  in  the  appraisal  may  provide  data  for  every  model  element 
included;  this  aspect  of  sampling  is  also  described  in  the  appraisal 
input. 


The  use  of  a  clearly  defined  appraisal  input  is  one  of  the  hallmarks  of 
SCAMPI  appraisals.  This  artifact  establishes  a  well-defined  basis  for 
performing  appraisal  activities. 

For  external  audits,  laws,  regulations,  or  contractual  restrictions  may 
limit  the  availability  of  certain  information  based  on  sensitivities 
associated  with  the  appraisal.  This  may  lead  to  creation  of  a  baseline 
appraisal  input  document  initially,  with  some  information  missing.  As 
the  information  becomes  available,  the  input  is  updated  and  reapproved 
by  the  sponsor.  The  sponsor,  in  this  context,  is  likely  to  be  a 
responsible  manager  from  an  external  organization,  and  not  a  senior 
site  manager  from  one  of  the  organizations  appraised.  The  senior  site 
managers  are  stakeholders  in  the  appraisal  process,  but  they  should  not 
be  treated  as  appraisal  sponsors  in  this  context. 


Con  tin  ued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Guidance:  Linkage 
Between  Appraisal 
Objectives  and 
Business  Goals 


Guidance  for  the 
Internal  Change  Agent 


The  linkage  between  the  business  goals  that  drive  choices  in  an 
organization’s  strategy,  and  the  role  to  be  played  by  the  appraisal  in 
furthering  those  goals,  must  be  well  understood.  The  sponsor’s 
rationale  for  conducting  an  appraisal  is  typically  a  very  important  part 
of  the  information  sought  by  appraisal  participants. 

By  documenting  the  purpose  of  the  appraisal  and  its  alignment  to  the 
business  goals  of  the  organization,  the  appraisal  (team)  leader 
frequently  serves  to  make  this  alignment  more  apparent  in  the 
organization.  In  addition,  the  communication  between  the  appraisal 
(team)  leader  and  the  sponsor  can  encourage  greater  understanding  of 
the  set  of  tactical  actions  sponsored  in  pursuit  of  business  goals. 

This  dialog  between  the  sponsor  and  the  appraisal  (team)  leader  at  the 
very  start  of  appraisal  planning  frequently  influences  sampling 
strategies  for  projects  and  individuals.  For  example,  consider  an 
organization  that  has  a  set  of  legacy  projects  based  on  older 
technology,  and  a  set  of  leading  edge  projects  based  on  new  (perhaps 
unproven)  technology.  The  choice  to  include  one  or  the  other  (or  both) 
types  of  projects  should  be  driven  by  the  anticipated  benefits — which 
might  be  unique  to  these  two,  potentially  very  different,  contexts. 

As  another  example,  consider  an  acquisition  setting  where  a  large 
number  of  bidders  are  expected  to  respond.  The  sponsor  might  choose 
a  phased  approach  in  which  SCAMPI  C  appraisals  are  used  to  “down- 
select,”  followed  by  a  smaller  number  of  SCAMPI  B  appraisals  to  be 
used  in  making  a  final  selection.  The  business  goals  in  this  context 
may  also  influence  the  model  scope  of  the  appraisal  as  well  as  the  type 
of  outputs  needed. 


Use  of  the  draft  appraisal  plan  as  the  vehicle  for  documenting  and 
obtaining  commitment  to  the  appraisal  input  will  be  more  feasible  in 
this  environment.  The  internal  change  agent  may  have  more 
opportunity  to  interact  with  the  sponsor  on  review  and  approval  of 
documents,  allowing  successive  versions  of  the  same  document  to  be 
used  as  the  planning  continues.  The  ability  to  “get  something  on  the 
manager’s  desk”  is  greater  through  internal  channels,  which  might  not 
be  available  to  the  professional  consultant  or  the  external  auditor. 


Con  tin  ued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Guidance  for  the 
Internal  Change  Agent 
(continued) 


As  a  member  of  the  organization,  the  internal  change  agent  may  also 
be  more  able  to  identify  subtle  issues  relating  to  appraisal  objectives. 
This  deeper  insight  will  also  inform  choices  about  scoping  the 
appraisal.  When  the  internal  change  agent  is  able  to  select  parts  of  the 
model  that  will  help  bring  attention  to  successful  improvement  efforts 
and  help  motivate  sponsorship  for  solving  other  challenges,  he  or  she 
may  be  most  qualified  to  focus  this  type  of  effort.  In  addition,  past 
experience  interacting  with  members  of  the  organization  will 
contribute  to  more  strategic  sampling  and  selection  of  interview 
participants. 

The  following  sampling  strategies  may  be  useful  in  this  context: 

•  uneven  sampling  of  model  detail,  where  some  process  areas  or 
goals  are  covered  with  much  more  detailed  data  while  others  are 
covered  only  at  a  high  level 

•  selection  of  Generic  Practices  as  the  focus  of  the  appraisal 

•  examination  of  integrated  product  and  process  development 
(IPPD)  content  from  the  CMMI  models  that  include  IPPD,  with 
limited  inclusion  of  the  rest  of  the  model 

•  a  focus  on  supplier  agreement  management  alone 

•  a  focus  on  an  individual  process  area  category  (e.g.,  Project 
Management,  or  Engineering) 

The  internal  change  agent  works  within  the  same  system  of  rewards 
and  incentives  (and  disincentives)  in  which  the  process  improvement 
program  exists.  The  internal  change  agent  is  frequently  called  upon  to 
convince  stakeholders  that  the  goals  of  the  appraisal  process  are  not  in 
conflict  with  the  goals  of  the  improvement  program.  In  many 
organizations,  the  availability  of  key  resources  is  a  major  issue.  An 
internal  change  agent  will  often  test  the  bounds  of  sponsorship  (out  of 
necessity)  to  get  the  “right  people”  involved  in  the  appraisal.  A 
professional  consultant  may  have  a  better  bargaining  position  in  some 
cases. 

The  influence  of  corporate  and  local  initiatives  sometimes  leads  to 
differences  in  practices  used  in  prior  projects.  This  will  affect  the 
sampling  choices,  because  too  much  variation  within  a  small  sample 
will  make  it  very  difficult  to  aggregate  information.  Responding  to 
sponsors’  interests  in  validating  the  return  from  previous  improvement 
efforts — or  providing  recommendations  about  an  ongoing  (or  future) 
effort — may  influence  the  scope  and  schedule  of  the  appraisal. 
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1.1  Analyze  Requirements  (continued) 


Guidance  for  the 

Professional 

Consultant 


The  professional  consultant  must  possess  competencies  that  enable 
him/her  to  quickly  learn  the  business  objectives  that  drive  the  need  for 
an  appraisal,  and  to  then  tailor  the  method  to  accommodate  the  context 
in  which  it  will  be  used.  Efficient  ways  to  gather  information  and  avoid 
miscommunication  are  needed.  The  professional  consultant  may  never 
get  the  opportunity  to  understand  the  full  detail  of  the  managerial, 
cultural,  and  political  systems  that  govern  the  organization.  This  has 
both  positive  and  negative  implications  for  people  who  choose  this  line 
of  work. 

The  contracting  process  for  consulting  services  will  often  include 
provisions  for  executing  a  non-disclosure  agreement,  and  it  may 
require  other  actions  that  potentially  delay  the  work  or  expand  the 
effort  required  to  perform  it.  Estimating  without  considering  these 
issues  may  lead  to  significant  problems. 

Documenting  a  separate  appraisal  input,  prior  to  providing  a  draft 
appraisal  plan  to  the  sponsor,  can  support  a  more  explicit  commitment 
process — which  can  support  maintenance  of  sponsorship  if  it  should  be 
called  into  question.  This  document  as  well  as  the  appraisal  plan  can  be 
identified  as  deliverables  and  treated  more  formally  in  this  context  than 
in  others. 

The  professional  consultant  typically  offers  a  variety  of  services  in 
addition  to  process  appraisals.  Organizations  often  hire  a  consultant  for 
improvement  support,  with  appraisal  services  being  a  relatively  small 
part  of  the  work.  The  professional  consultant  frequently  plans 
appraisals  to  support  the  generation  of  data  and  information  in  the 
context  of  an  ongoing  program  of  training  and  consultation.  While  the 
internal  change  agent  may  have  better  access  to  information  and 
insights  that  support  this  integration,  the  professional  consultant  will 
have  a  broader  frame  of  reference  from  his/her  experience  of  working 
on  improvement  programs  in  other  client  organizations. 

A  purely  education-oriented  appraisal  is  a  common  service  offered  by 
professional  consultants.  Using  a  workshop  format,  organizations  can 
perform  a  gap  analysis  or  receive  focused  training  on  some  aspect  of 
process  or  process  improvement.  The  SCAMPI  C  method  supports 
many  different  approaches  like  this. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Guidance  for  the 
External  Auditor 


The  sponsor  for  an  external  audit  may  be  from  an  external  organization 
and  not  a  senior  manager  at  the  site(s)  where  appraisal  activities  are 
carried  out.  Some  of  the  information  required  in  the  appraisal  input 
(such  as  process  context  and  organizational  unit)  that  could  otherwise 
be  provided  by  the  sponsor  must  be  collected  from  the  organization 
being  appraised. 

The  external  auditor  may  work  in  the  context  of  a  government  or 
corporate  appraisal.  In  either  case,  there  may  be  multiple  appraisal  on¬ 
site  periods  being  planned,  with  a  sponsor  who  represents  an  external 
authority  to  the  organization(s)  being  appraised.  Planning  many  site 
visits  with  an  overarching  appraisal  input  and/or  appraisal  plan  can  be 
an  effective  strategy  for  making  the  best  use  of  the  sponsor’s  authority. 

There  are  frequently  numerous  and  diverse  stakeholders  to  the 
performance  of  appraisals  in  this  context.  Relevant  stakeholders  may 
include  the  sponsor,  program  manger,  contracting  officer,  and 
personnel  responsible  for  the  acquisition  and  source  selection. 

If  the  appraisal  is  part  of  outsourcing  or  acquisition  decision-making, 
analyzing  requirements  should  occur  early.  The  SCAMPI  planning 
activities  should  be  integrated  with  the  other  activities  and  scheduled 
accordingly. 

It  may  be  more  challenging  to  devise  sampling  strategies  for  projects 
and  appraisal  participants  in  this  context.  Organizations  teaming  to  bid 
on  a  contract,  or  virtual  organizations  being  monitored  over  time, 
present  special  challenges  to  the  collection  and  reporting  of  data  that 
best  represents  the  process  of  interest.  For  supplier  selection  uses,  the 
connection  between  the  people  and  projects  appraised,  and  the  “to  be 
formed”  project,  is  sometimes  very  difficult  to  establish.  Identifying 
the  specializations  and  skill  levels  needed  on  the  future  project,  and 
assuring  that  the  appraisal  includes  the  “right  people,”  is  often  the 
strategy  to  employ,  unless  the  team  is  already  intact. 

Using  a  SCAMPI  C  to  “down-select”  from  a  larger  group  of  potential 
suppliers,  then  following  with  focused  SCAMPI  B  appraisals,  can  be 
an  effective  strategy  for  managing  appraisal  resources  when  the 
contract  under  consideration  is  of  sufficient  magnitude. 

The  intent  to  use  appraisal  results  in  the  context  of  an  acquisition  or 
contractual  situation  will  place  requirements  on  the  nature  and  content 
of  the  appraisal  outputs.  These  matters  must  be  understood  and 
communicated  to  stakeholders,  such  as  appraisal  team  members, 
appraisal  participants,  and  recipients  of  appraisal  outputs. 
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1.2  Develop  Appraisal  Plan 


Overview 


Required  Practices 

ARC.4.1.2.b 


Parameters  and  Limits: 
General 

ARC.4.3.5 


Parameters  and  Limits: 
Required  Contents  of 
Appraisal  Plan 

ARC.4.2.12 

ARC4.3.5.a-e 


Parameters  and  Limits: 

Sponsor 

Responsibilities 

ARC  4.1.1 
ARC  4.2.9 

Parameters  and  Limits: 
Review  of  Appraisal 
Plan 


An  appraisal  plan  forms  the  basis  for  realistic  commitments  and  serves 
as  the  basis  for  monitoring  and  controlling  the  appraisal  process. 
Therefore,  both  uses  of  the  plan  (establishing  commitments  and 
monitoring  the  process)  are  addressed  in  the  requirements  for 
developing  an  appraisal  plan. 


The  appraisal  team  leader  shall 

•  Document  and  maintain  the  appraisal  plan 

•  Obtain  sponsor  approval  of  the  appraisal  plan 


All  required  elements  of  the  appraisal  plan  must  be  included  in  the 
final  version  of  the  appraisal  plan.  Preliminary  versions  that  lack 
individual  items  (e.g.,  logistical  details)  may  be  used  to  manage  the 
commitment  process — especially  if  the  process  of  documenting  the 
appraisal  plan  must  begin  prior  to  the  designation  of  an  appraisal 
(team)  leader. 

Final  approval  of  the  appraisal  plan  must  be  designated  by  the  sponsor, 
who  must  indicate  his/her  approval  by  signing  the  document.  The  team 
leader  must  also  sign  the  plan  to  indicate  his/her  understanding  and 
commitment.  This  sign-off  must  occur  before  any  interviews  or  artifact 
reviews  may  begin. 

Electronic  signatures  may  be  used  as  long  as  the  same  parameters  and 
limits  specified  for  electronic  signatures  in  Process  1.1  are  met. 


At  a  minimum,  the  appraisal  plan  must  include  the  following: 

1 .  the  appraisal  input  (by  reference  if  desired) 

2.  the  activities  to  be  performed  in  conducting  the  appraisal 

3.  resources  and  schedule  assigned  to  appraisal  activities 

4.  appraisal  logistics 

5.  risks  and  mitigations  to  appraisal  planning  and  execution 


The  appraisal  sponsor  is  responsible  for  assuring  that  the  appraisal 
input  is  accurate  and  that  the  appraisal  plan  includes  the  appropriate 
information  as  he/she  reviews  and  approves  these  documents.  The 
sponsor  must  ensure  that  the  commitments  made  during  planning  are 
realistic  and  that  they  are  met. 


If  an  appraisal  team  is  used,  each  member  of  the  team  must  be 
provided  an  opportunity  to  review  the  plan  prior  to  its  approval  by  the 
appraisal  sponsor.  The  sponsor  may  also  designate  other  people  to 
review  the  plan  as  he/she  deems  appropriate. 
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1.2  Develop  Appraisal  Plan  (continued) 


Parameters  and  Limits: 
Revision  of  the 
Appraisal  Plan 


Parameters  and  Limits: 
Appraisal  Plan 
Template 


Parameters  and  Limits: 
Risk  Management 


The  appraisal  plan  is  baselined  when  the  appraisal  sponsor  signs  it  to 
reflect  her/his  commitment  to  the  appraisal  reflected  in  the  plan.  If  and 
when  any  element  of  the  appraisal  input  must  change,  the  appraisal 
plan  must  be  re-baselined  through  review  and  sign-off  on  a  revised 
version  of  the  plan  by  the  sponsor  and  appraisal  (team)  leader.  The 
evolution  of  the  plan  must  be  documented.  This  is  typically  done  using 
a  “change  history  page”  within  the  plan. 


An  appraisal  plan  template  is  available  in  the  materials  provided  by  the 
SEI.  Use  of  this  template  is  strongly  recommended.  One  might  choose 
not  to  use  this  template  if  there  is  a  plan  template  in  wide  use  in  the 
organization  and  use  of  that  template  would  result  in  more  effective 
planning  and  monitoring. 


Appraisal  planning  must  address  any  risks  to  the  appraisal  in  order  to 
be  considered  complete.  It  is  not  sufficient  to  merely  identify 
perfunctory  issues,  such  as  resource  constraints  and  the  possibility  that 
a  team  member  might  take  ill,  which  are  typical  and  generally 
mitigated  with  general  contingency  planning. 

It  may  also  be  important  to  identify  and  mitigate  risks  that  pertain  to 
planning.  Where  large  numbers  of  people  must  participate  in  planning, 
or  where  potential  barriers  to  obtaining  and/or  providing  important 
information  exist,  risks  and  mitigation  strategies  for  effective  appraisal 
planning  must  be  documented. 


Con  tin  ued  on  next  page 
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1.2  Develop  Appraisal  Plan  (continued) 


Guidance:  Risk 
Management 


Appraisal  planning  must  address  any  risks  to  the  appraisal  in  order  to 
be  considered  complete.  It  is  not  sufficient  to  merely  identify 
perfunctory  issues,  such  as  resource  constraints  and  the  possibility  that 
a  team  member  might  take  ill,  which  are  typical  and  generally 
mitigated  with  general  contingency  planning. 

It  may  also  be  important  to  identify  and  mitigate  risks  that  pertain  to 
planning.  Where  large  numbers  of  people  must  participate  in  planning, 
or  where  potential  barriers  to  obtaining  and/or  providing  important 
information  exist,  risks  and  mitigation  strategies  for  effective  appraisal 
planning  must  be  documented. 

Effective  risk  management  planning  addresses  key  assumptions  on 
which  the  success  of  the  appraisal  is  based.  For  example:  “The 
appraisal  on-site  period  will  take  place  at  the  ABC  Company  campus 
from  <specified  dates>”.  This  statement  in  an  appraisal  plan  presumes 
the  given  facilities  are  available  and  the  dates  are  open  for  all 
participants.  Questions  to  ask  include:  Is  the  facility  available  and  have 
the  appropriate  rooms  been  set  aside  for  participants  and  team 
members?  Do  the  dates  cited  encompass  any  national  holidays, 
company  events,  or  major  events,  such  as  a  program  review  of  a  key 
participant?  If  positive  responses  to  the  questions  are  not  forthcoming, 
then  mitigation  and  contingency  planning  should  be  invoked  and 
documented  in  the  appraisal  plan.  Similarly,  involvement  of  personnel 
with  particular  types  of  capabilities  or  information  is  an  area  where  risk 
mitigation  planning  is  frequently  inadequate.  Risks  that  simply  state 
“key  participants  may  be  unavailable”  are  not  adequate  without 
documenting  the  context — for  example,  “key  participants  may  be 
unavailable  due  to  concurrent  ISO  audit,  which  cannot  be 
rescheduled.”  Documenting  the  needs  to  be  fulfilled  by  such  people 
will  be  necessary  in  order  to  find  replacements,  in  the  event  the  risk  is 
realized  and  becomes  a  problem. 

Contingency  planning,  by  adding  buffer  time  to  the  schedule  or 
identifying  alternate  participants  for  key  (or  all)  interview  participants, 
is  also  an  element  of  risk  management  planning. 
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1.2  Develop  Appraisal  Plan  (continued) 


Guidance: 

Relevant  Stakeholders 


Guidance  for  the 
Internal  Change  Agent 


The  purpose  of  documenting  a  plan  includes  forecasting  the  future  to 
set  expectations,  as  well  as  establishing  a  basis  for  determining 
corrective  actions  if  reality  departs  from  initial  expectations.  The 
involvement  of  relevant  stakeholders  in  drafting  and  reviewing  the  plan 
facilitates  their  participation  and  helps  to  assure  their  buy-in  if 
corrective  actions  are  needed.  The  following  role  names  describe 
people  who  are  typically  considered  relevant  stakeholders  for  the 
appraisal  plan: 

•  appraisal  sponsor 

•  appraisal  team  leader 

•  appraisal  team  members 

•  administrative  staff  supporting  on-site  activities 

•  sponsor-designated  personnel  in  the  organizational  unit 

•  sponsor-designated  acquisition  agents 


For  the  internal  change  agent,  the  appraisal  typically  plays  a  role  in  an 
ongoing  program  of  sponsored  change.  Given  this  backdrop,  the 
appraisal  plan  can  be  an  asset  to  be  used  to  reinforce  a  coherent  image 
of  sponsorship.  The  linkage  between  business  goals  and  appraisal  goals 
documented  in  the  plan  can  be  a  touchstone  for  members  of  the 
organization,  and  may  need  to  be  written  to  be  consistent  with  other 
published  information  in  the  organization. 

The  contents  of  the  plan  may  also  include  more  of  the  pre-  and  post¬ 
appraisal  activities  than  would  otherwise  be  typical  for  appraisals  led 
by  people  external  to  the  organization.  The  internal  change  agent  may 
also  include  more  sponsorship-building  activities  as  part  of  the 
appraisal  as  well.  Opportunities  to  have  “all-hands  meetings”  or  other 
activities  to  facilitate  communication  during  the  appraisal  may  be 
desirable. 

Finally,  risk  and  risk  mitigation  planning  may  be  influenced  by  similar 
planning  associated  with  the  larger  improvement  program.  The 
implication  that  risks  arising  during  the  appraisal  can  affect  the  success 
of  the  process  improvement  program  can  be  articulated.  Mitigation 
strategies  to  address  risks  can  be  documented  in  the  appraisal  plan  to 
elevate  their  visibility  to  the  sponsor,  who  commits  to  the  plan  (and  the 
risk  mitigation  strategies)  by  signing  the  document. 
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1.2  Develop  Appraisal  Plan  (continued) 


Guidance  for  the 

Professional 

Consultant 


The  professional  consultant  may  manage  relationships  and  planning 
using  a  variety  of  documents  in  addition  to  the  appraisal  plan. 
Frequently  there  is  an  overarching  contractual  document  that  lists  the 
appraisal  plan  as  one  of  the  deliverables.  It  is  rare  that  the  appraisal 
plan  is  the  only  document  used  to  establish  sponsorship  and 
commitment  to  the  appraisal. 

The  distinction  between  the  appraisal  input  and  the  appraisal  plan  may 
be  more  pronounced  for  the  professional  consultant.  The  availability  of 
key  information  to  plan  the  appraisal  may  be  very  limited  in  the  early 
stages  of  planning.  Therefore,  creating  a  separate  appraisal  input 
document  may  help  to  establish  commitment  early  on,  without  having 
to  wait  for  more  complete  information.  In  addition,  the  possibility  that 
changes  will  be  needed  as  new  information  is  obtained  is  greater  for 
the  professional  consultant,  who  must  frequently  rely  on  people  she/he 
does  not  know  very  well. 

Because  the  professional  consultant  does  not  typically  work  in  the 
appraised  organization,  her/his  familiarity  with  the  location,  culture, 
and  policies  of  the  organization  may  be  limited.  Logistics  planning,  as 
well  as  provisions  for  security  (e.g.,  badging,  access  to  electronic 
media,  and  the  ability  to  receive  documents  in  advance)  may  require 
more  attention  in  the  appraisal  plan. 

Risk  planning  often  takes  on  added  importance,  as  the  ability  to  spend 
adequate  time  examining  data  and  information  in  advance  may  be 
limited.  Risk  of  misconceptions  regarding  objective  evidence  mapping, 
or  failure  to  identify  key  personnel  to  participate  in  the  appraisal,  may 
necessitate  more  readiness  reviews  as  well  as  much  more  explicit 
elaboration  in  the  appraisal  plan.  Strategies  to  prioritize  appraisal 
goals,  develop  contingency  plans  to  reduce  the  appraisal  scope,  and 
build  extra  reserves  for  time  and  personnel  can  be  very  important. 
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1.2  Develop  Appraisal  Plan  (continued) 


Guidance  for  the 
External  Auditor 


The  external  auditor  may  be  required  to  satisfy  requirements  about  the 
nature  of  information  she/he  is  permitted  to  reveal  to  the  organization, 
as  well  as  requirements  that  limit  access  to  the  organization.  The 
validation  of  preliminary  findings  may  be  constrained  to  one-way 
communication  using  documents  rather  than  a  “live”  presentation.  The 
appraisal  plan  must  consider  solicitation  updates,  a  strategy  for 
providing  feedback  on  findings  through  discussions  or  other 
mechanism,  and  a  strategy  for  delivering  final  findings  post  award. 

When  the  appraisal  is  used  to  monitor  performance  on  an  existing 
contract,  the  appraisal  plan  may  become  an  element  of  the  contract.  It 
may  be  necessary  to  address  the  strategy  for  conducting  appraisals 
across  the  development  life  cycle  as  part  of  risk  mitigation.  The 
relevance  of  contractual  implications  associated  with  the  appraisal  may 
be  addressed  or  referenced  in  the  appraisal  plan  as  well. 

Many  of  the  risks  described  in  relation  to  the  professional  consultant 
may  apply  as  well.  However,  for  the  external  auditor,  there  may  be 
more  limited  flexibility  in  re-planning  events  or  making  other  mid¬ 
course  corrections.  The  appraisal  goals  of  the  external  appraisal 
sponsor  may  not  always  be  perceived  as  benevolent  to  appraisal 
participants.  Planning  must  ensure  that  interactions  during  the 
appraisal  yield  the  necessary  outcomes  without  undue  perturbation  to 
the  business  of  the  organization  being  appraised. 
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1.3  Select  and  Prepare  Team 


Overview 


Required  Practices 

ARC.4.1.2.d 


Parameters  and  Limits: 
Appraisal  (Team) 
Leader  Qualifications 

ARC.4.2.6.a-c 

Parameters  and 
Limits:  Team  Member 
Qualifications 

4.2.5.a-c 

4.2.7 


Use  of  an  appraisal  team  is  required  for  SCAMPI  B,  but  not  required 
for  SCAMPI  C.  However,  as  a  team-based  activity,  appraisals  gain  a 
great  deal  of  momentum  from  the  participation  of  people  with  diverse 
perspectives  and  experience.  Use  of  a  team  in  the  performance  of  a 
SCAMPI  C  is  recommended  in  contexts  where  there  is  sufficient 
workload  and  support  to  justify  the  expenditure. 


If  performing  a  SCAMPI  B,  the  appraisal  team  leader  shall  select  team 
members  who  meet  the  criteria — individually  and  as  a  group — 
specified  in  the  sections  on  qualifications,  below. 

If  using  a  team  in  performing  a  SCAMPI  C,  the  information  specified 
below  shall  be  documented.  Although  the  minimums  may  not  be  met, 
the  team  leader  must  specify  the  actual  experience  levels  present  on  the 
team. 

If  using  a  team,  the  appraisal  team  leader  shall  provide  training  in 
accordance  with  the  requirements  specified  below. 


The  CMMI  Steward  maintains  a  program  of  qualification,  training,  and 
monitoring  for  SCAMPI  Lead  Appraisers.  Information  relating  to  the 
specific  qualifications  required  of  appraisal  (team)  leaders  is  published 
on  the  SETs  Web  site  at  http://www.sei.cmu.edu. 


The  minimum  acceptable  team  size  for  a  SCAMPI  B  appraisal  is  two 
people,  including  the  team  leader.  SCAMPI  C  does  not  require  a  team. 

In  selecting  team  members  for  a  SCAMPI  B,  the  following  criteria 
must  be  met: 


Experience  Category 

Individuals 

Team 

Engineering  Discipline 
(each  discipline  in  scope) 

At  least  5  years 
on  average 

At  least  10  years 
total 

Management 

At  least  one  with 

5  years 

At  least  5  years 
total 

Each  life-cycle  phase  used  by 
the  organizational  unit 

The  team  must  include  experienced 
practitioners  for  at  least  the  majority  of 
life-cycle  phases  in  use. 

Completion  of  the 

Introduction  to  CMMI  course 
taught  by  an  SEI-authorized 
instructor 

Each  team  member  must  successfully 
complete  model  training  prior  to 
participating  in  the  appraisal. 

Con  tin  ued  on  next  page 
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1.3  Select  and  Prepare  Team  (continued) 


Parameters  and  Limits: 
Documenting  Team 
Member  Qualifications 


Parameters  and  Limits: 
Prerequisite  Training 
for  Team  Members 


Parameters  and  Limits: 
Team  Training 
Requirements 


A  table  (or  other  form  of  documentation)  summarizing  the  satisfaction 
of  minimum  requirements  (for  SCAMPI  B)  or  the  actual  level  of 
experience  on  the  team  (for  SCAMPI  C)  must  be  included  in  the 
appraisal  input,  per  the  Parameters  and  Limits  specified  in  Process  1.1: 
Analyze  Requirements. 


Every  team  member  must  complete  the  required  model  training;  this  is 
a  uniform  requirement  for  all  SCAMPI  methods.  To  serve  as  a  team 
member  on  a  SCAMPI,  one  must  successfully  complete  the 
Introduction  to  CMMI  course.  This  course  must  be  taught  by  an  SEI- 
authorized  instructor  working  on  behalf  of  an  SEI  Partner  in  order  to 
be  accepted. 


The  team  training  materials  used  by  SCAMPI  Lead  Appraisers  are 
expected  to  be  tailored  and  customized,  as  appropriate,  for  use  in 
preparing  teams  for  a  given  appraisal.  Depending  on  the  scope  of  the 
appraisal,  and  the  level  of  responsibility  assigned  to  the  team  members, 
different  subsets  of  the  standard  team  training  are  to  be  used. 

At  a  minimum,  all  team  members  must  be  trained  on  the  following 
topics  using  information  from  the  SCAMPI  B  team  training  materials 
provided  by  the  SEI: 

•  SCAMPI  method  overview 

•  appraisal  planning,  including  the  contents  of  the  appraisal  plan 

•  objective  evidence  collection  and  analysis 

•  team  decision  making 

•  appraisal  confidentiality  and  non-attribution 

•  practice  characterization 

•  findings  development,  verification,  and  validation 

•  appraisal  output  requirements 

For  teams  involved  in  U.S.  government  source  selection  or  contract 
monitoring  appraisals,  team  members  must  also  be  trained  in 

•  applicable  laws,  regulations,  and  policies  that  affect  the  appraisal, 
such  as  Federal  Acquisition  Regulations,  DoD  service,  or 
organization  regulations  and  policies 

•  role  of  the  appraisal  and  the  appraisal  team  in  the  source  selection 
or  contract  monitoring  processes  and  structures 

•  limitations  on  findings  development,  validation,  and  release 

•  special  domain  and/or  model  requirements  (e.g.,  space,  C21,  IT: 
supplier  sourcing,  statistical  process  management) 
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1.3  Select  and  Prepare  Team  (continued) 


Parameters  and  Limits: 
Team  Member  Roles 
and  Responsibilities 

ARC  4.2.8 


Guidance:  Use  of 
Mini-Teams 

ARC  4.2.8 


Guidance  for  the 
Internal  Change  Agent 


The  appraisal  team  leader  (when  a  team  is  used)  is  responsible  for 
defining  specific  roles  and  responsibilities  for  each  member  of  the 
team.  Balancing  the  workload  of  appraisal  activities  and  matching 
individual  tasks  to  the  skills  and  talents  of  the  team  members  are 
essential  responsibilities  of  the  team  leader. 


In  appraisals  with  a  substantial  model  or  organizational  scope, 
appraisal  team  leaders  frequently  divide  the  appraisal  team  into  mini¬ 
teams  in  order  to  distribute  responsibility  evenly  across  the  team.  This 
strategy,  if  used,  must  be  made  clear  to  team  members  as  they  are 
trained  in  how  to  fulfill  their  specific  responsibilities. 

Most  often,  each  mini-team  is  assigned  a  subset  of  the  process  areas  in 
the  scope  of  the  appraisal,  for  them  to  take  primary  responsibility  for 
data  collection  and  initial  analysis.  Later  the  larger  team  reviews  the 
results  of  the  work  of  each  mini-team. 

In  other  situations,  mini-teams  are  assigned  to  focus  on  individual 
projects  included  in  the  appraisal.  This  arrangement  is  particularly 
beneficial  when  the  number  of  projects  included  in  the  appraisal 
exceeds  the  number  of  process  areas  included  in  the  appraisal. 


It  is  important  for  the  internal  change  agent  to  select  team  members 
who  are  considered  credible  by  the  organization.  The  people  selected 
are  typically  proponents  of  process  improvement  from  different  parts 
of  the  organization.  Members  of  an  engineering  process  group  are  very 
often  selected  as  team  members.  Managers  and  other  people  with 
authority  over  a  large  number  of  the  interview  participants  are 
frequently  avoided  because  their  participation  may  influence  the 
interaction  among  appraisal  participants  in  unproductive  ways,  though 
a  strict  prohibition  against  their  involvement  may  not  always  make 
sense. 

In  selecting  the  appraisal  team  members,  consider  selecting  individuals 
who  are  familiar  with  the  participating  projects,  have  a  working 
knowledge  of  the  organizational  standard  process,  and  can  easily 
access  or  locate  additional  artifacts.  Drawing  from  a  pool  of 
established  experts  who  are  accustomed  to  working  together  limits  the 
need  for  extensive  team  training  and  provides  savings  over  time  for 
organizations  that  conduct  frequent  appraisals. 
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1.3  Select  and  Prepare  Team  (continued) 


Guidance  for  the 

Professional 

Consultant 


The  professional  consultant  will  frequently  mentor  one  or  more  points 
of  contact  from  the  organization  to  aid  in  the  transfer  of  knowledge. 
This  person  (or  people)  will  facilitate  access  to  information  and  people 
in  the  organization.  The  type  of  coaching  and  mentoring  they  receive 
will  often  be  more  detailed  and  time-consuming  than  for  other 
members  of  the  team.  Team  members  from  within  the  organization  are 
often  selected  on  the  basis  of  their  familiarity  with  the  operating 
procedures  of  the  organization  and  the  projects  involved  in  the 
appraisal,  as  well  as  their  ability  to  locate  information  needed  to 
perform  the  appraisal. 

Training  offerings  may  be  elaborate  or  very  streamlined,  depending  on 
the  focus  of  the  consultant  and  the  nature  of  his  or  her  agreement  with 
the  customer.  Organizations  sometimes  use  consultants  for  a  series  of 
training  and  mentoring  activities  in  which  the  appraisal  is  only  one 
event.  The  goal  of  the  overall  engagement  is  often  to  build  “organic 
capability”  in  the  organization  for  using  the  tools  and  techniques 
employed  by  the  professional  consultant. 

As  a  team  leader  from  outside  the  organization,  the  professional 
consultant  must  gain  familiarity  with  the  relevant  characteristics  of  the 
organization  in  order  to  make  reasonable  interpretations  of  CMMI. 
This  process  can  include  presentations  and  interviews  during  the 
training  activities,  especially  if  team  members  external  to  the 
organization  are  used.  The  professional  consultant  might  also  receive  a 
briefing  to  become  acquainted  with  the  organizational  unit’s 
characteristics  and  documentation,  the  structure  of  the  organization’s 
standard  process,  and  descriptions  of  the  projects  involved.  This 
increases  the  consultant’s  understanding  of  the  context  of  the  appraisal. 
Learning  the  roles  and  responsibilities  deployed  in  the  organization  is 
typically  addressed  as  well. 

It  is  often  critical  that  training  activities  adequately  demonstrate  and 
explain  data  collection  and  management  tools  to  be  used. 
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1.3  Select  and  Prepare  Team  (continued) 


Guidance  for  the 
External  Auditor 


Appraisals  led  by  external  auditors  are  sometimes  part  of  a  series  of 
appraisals  in  which  multiple  organizations  are  examined.  In  some 
situations,  multiple  teams  may  work  in  parallel  or  in  succession  to 
collect  information  that  will  be  used  to  monitor  ongoing  activities  or  to 
make  a  decision  that  affects  multiple  organizations.  Also,  there  are 
sometimes  significant  financial  implications  for  the  participants  in  the 
appraisal(s).  Single-person  SCAMPI  C  appraisals  are  not  likely  to  be 
appropriate  for  many  situations  where  an  external  auditor  is  required. 

The  appraisal  team  members  frequently  receive  a  briefing  to  acquaint 
them  with  the  organizational  unit’s  characteristics  and  documentation 
and  the  structure  of  the  organization’s  standard  process  as  well  as 
descriptions  of  the  projects  involved.  This  increases  the  team’s 
understanding  of  the  context  of  the  appraisal.  Learning  the  roles  and 
responsibilities  deployed  in  the  organization  is  typically  addressed  as 
well. 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence 


Overview  One  hallmark  of  all  SCAMPI  appraisals  is  an  emphasis  on  the 

structured  use  of  objective  evidence  acquired  early  in  the  appraisal 
process.  The  important  distinction  between  the  “discovery”  and 
“verification”  approaches  to  appraisal  provides  a  basis  for 
differentiating  the  efficiency  of  the  appraisal  process. 

A  documented  set  of  Practice  Implementation  Indicators  (PIIs)  is  not  a 
required  input  for  the  SCAMPI  family  of  methods.  However, 
efficiency  is  greatly  enhanced  if  objective  evidence  can  be  identified  in 
advance  and  then  associated  with  elements  of  the  organization  as  well 
as  components  of  the  model.  The  availability  of  such  key  information 
shapes  the  data  collection  plan,  and  it  is  typically  discussed  with 
personnel  who  are  supporting  the  appraisal  activities  at  the  site  where  it 
is  being  conducted.  The  formality  with  which  this  process  is  conducted 
must  be  matched  to  the  nature  of  the  data  collection  envisioned  for  the 
appraisal. 

Finally,  preparing  the  members  of  the  organization  who  will  participate 
in  the  appraisal  tends  to  enhance  the  success  of  efforts  aimed  at  getting 
objective  evidence  in  advance.  Communication  mechanisms  used  in 
preparing  participants  range  from  formal  presentations  to  brief  email  or 
voicemail  messages.  Appropriate  levels  of  detail  regarding  the  data 
structure,  described  below,  will  need  to  be  made  clear  to  different 
appraisal  participants. 


Required  Practices  The  appraisal  team  leader  shall 

•  Establish  and  communicate  a  strategy  for  preparing  appraisal 
participants  to  provide  needed  information 

•  Communicate  with  appraisal  participants  to  establish  expectations 
and  answer  questions 

•  Obtain  an  initial  set  of  objective  evidence 

•  Inventory  the  objective  evidence 


Continued  on  next  page 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence  (continued) 


Parameters  and  Limits:  The  strategy  for  preparing  appraisal  participants  may  include  written 
Communication  with  communications,  formal  presentations,  and  direct  interaction  with 
Appraisal  Participants  individual  appraisal  participants  (one  on  one).  Information  about  the 
ARC.4.1.2.C  purpose  of  the  appraisal  and  the  roles  and  responsibilities  of 

ARC4.3.1.a-e  participants  must  be  provided  to  everyone  who  will  supply  data  to  the 

appraisal.  In  a  supplier  selection  application,  the  request  for  proposals 
(RFP)  or  other  similar  communications  should  be  used  as  an  element 
in  the  appraisal  communication  strategy. 

The  communications  used  to  prepare  participants  must  cover  the 
following  content: 

1 .  purpose  of  the  appraisal 

2.  scope  of  the  appraisal 

3.  appraisal  approach 

4.  roles  and  responsibilities  of  participants 

5.  schedule  of  appraisal  activities 

Every  participant  must  have  an  opportunity  to  understand  the  same 
basic  information  about  the  appraisal  and  must  have  a  way  to 
communicate  a  question  to  the  appraisal  (team)  leader. 


Parameters  and  Limits: 
Confidentiality  and 
Non-Attribution 
Provisions 

ARC4.2.15 


Agreements  with  the  appraisal  sponsor,  as  reflected  in  the  appraisal 
input,  must  be  included  in  communications  with  appraisal  participants. 
Depending  on  their  potential  impact,  this  topic  may  be  covered  during 
every  interaction  with  the  participants  of  an  appraisal. 


Continued  on  next  page 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence  (continued) 

As  a  data  structure,  the  use  of  Practice  Implementation  Indicators 
(PIIs)  is  required.  Specific  requirements  are 

•  the  classification  of  artifacts  into  direct  and  indirect  indicators  of 
practice  implementation 

•  labeling  affirmations  as  a  distinct  source  of  data 

•  associating  each  item  of  evidence  with  a  particular  practice  in  a 
CMMI  model  (or  a  non-model  category) 

•  associating  each  item  of  evidence  with  either  a  given  project  or  an 
organizational  function 

•  using  inventories  of  data  based  on  this  framework  to  establish  that 
sufficient  data  have  been  examined  to  support  appraisal  outcomes. 
Section  2.3  in  Parameters  and  Limits  provides  further  discussion 
on  this  topic. 

In  light  of  this  structure,  the  data  may  be  rather  incomplete  and  lacking 
in  detail  in  a  SCAMPI  C  appraisal  as  compared  to  SCAMPI  A. 
However,  it  is  expected  that  this  data  structure  will  always  be  used. 

Use  of  this  data  structure  in  reporting  outcomes  of  the  appraisal,  while 
strongly  encouraged,  is  not  mandatory. 

Parameters  and  Limits:  Initial  objective  evidence  may  be  as  much  as  a  complete  set  of  PII 
Obtaining  Initial  databases  with  supporting  documentation  integrated  using  hyperlinks, 

Objective  Evidence  or  as  little  as  a  document  that  maps  elements  of  CMMI  to  processes 

and/or  artifacts  used  in  the  organization.  For  SCAMPI  C,  the  initial 
objective  evidence  may  be  merely  a  set  of  notes  written  by  the 
appraisal  (team)  leader  during  a  telephone  conversation  with  the 
sponsor. 

While  the  initial  set  of  objective  evidence  need  not  conform  to  the  PII 
structure  described  above,  the  data  used  during  the  appraisal  process 
must  be  organized  in  this  way. 

Continued  on  next  page 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence  (continued) 

Parameters  and  Limits:  Objective  evidence  must  be  labeled  and  inventoried  in  order  to 
Inventory  of  Initial  determine  how  much  evidence  there  is  for: 

Objective  Evidence  •  each  practice  included  in  the  scope  of  the  appraisal 

•  each  project  included  in  the  scope  of  the  appraisal 

•  each  type  of  data  (i.e.,  direct  artifacts,  indirect  artifacts,  and,  where 
appropriate,  affirmations) 

The  amount  and  quality  of  objective  evidence  obtained  in  advance 
must  be  considered  in  preparing  for  future  activities,  also  taking  into 
account  the  minimum  amount  of  evidence  required  to  support  valid 
appraisal  outputs,  which  is  specified  later  in  this  document. 

The  inventory  must  be  sufficiently  well  documented  to  support 
detailed  planning  for  discovery  of  the  objective  evidence  still  needed  to 
meet  the  minimum  standards.  Documenting  these  results  in  the  format 
of  a  PII  database  is  recommended  but  not  required.  Such  a  structure 
(which  is  required  to  organize  data  later)  displays  the  different  types  of 
evidence  available  for  each  practice  and  for  each  project  (as 
appropriate)  included  in  the  scope  of  the  appraisal. 

It  is  not  expected  that  each  item  of  evidence  will  be  reviewed  for 
sufficiency  at  this  point.  However,  a  cursory  review  of  the  contents  of 
the  evidence  provided  will  help  identify  any  gross  misconceptions  that 
could  threaten  the  proper  interpretation  of  the  evidence.  A  much  more 
detailed  examination,  which  includes  a  focus  on  the  appropriateness  of 
individual  items,  is  carried  out  during  the  readiness  review  (described 
below).  At  this  stage  in  the  process,  the  inventory  should  focus  only  on 
applicability  and  suitability,  at  most,  and  not  on  sufficiency  of  the 
evidence  in  meeting  the  intent  of  the  model  practice. 

Guidance:  Opening  A  group  meeting  or  larger  presentation  frequently  provides  the  best 

Meeting  forum  for  communication  with  the  entire  set  of  participants.  This  type 

of  venue  is  desirable  because  it  provides  added  visibility  for  the 
activities  of  the  appraisal  and  the  support  of  the  management  in  the 
organization.  In  this  meeting,  participants  are  encouraged  to  ask 
questions  to  ensure  that  they  all  understand  the  expectations  for  their 
participation,  and  that  other  participants  can  hear  the  same  information 
in  response  to  these  questions. 

Continued  on  next  page 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence  (continued) 

Guidance  for  the  The  preparation  of  participants  for  the  appraisal  is  likely  to  be  one  of 

Internal  Change  Agent  the  ongoing  responsibilities  of  the  internal  change  agent.  Briefings  or 
other  communications  that  orient  participants  regarding  their  roles  and 
participation  may  be  part  of  an  ongoing  communication  strategy,  and 
they  may  require  more  coordination  and  advance  notification. 

Extensive  databases  and  document  libraries  may  be  under  the 
management  of  the  internal  change  agent  or  one  of  his  or  her  peers. 
The  internal  change  agent  frequently  works  in  the  organization  to 
amass  information,  index  it  appropriately,  and  review  it  in  advance  of 
an  appraisal.  Familiarity  with  the  data  that  forms  the  basis  for  the 
appraisal  can  provide  great  efficiency. 

Whether  working  to  perform  a  single  appraisal  or  delivering  services  to 
an  organization  on  an  ongoing  basis,  the  professional  consultant 
typically  employs  a  communication  strategy  for  preparing  participants. 
Information-sharing  briefings,  written  communications,  or 
teleconferences  may  be  used. 

As  an  “outsider”  the  professional  consultant  frequently  faces 
limitations  on  the  amount  of  data  and  insight  she  or  he  can  obtain  in 
advance  of  the  appraisal.  Classification  or  proprietary  restrictions  may 
also  limit  the  consultant’s  ability  to  take  materials  away  from  the 
organization’s  office  locations  for  review  in  advance  or  in  off-hours 
during  the  appraisal. 

Continued  on  next  page 
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1.4  Prepare  Participants  and  Obtain  Initial  Objective 
Evidence  (continued) 

Guidance  for  the  For  the  external  auditor,  preparing  participants  could  take  the  form  of 

External  Auditor  emails  describing  the  appraisal  process  or  a  briefing  on  the  first  day 

that  the  appraisal  (team)  leader  is  on  site.  The  appraisal  participants 
typically  require  a  vehicle  for  asking  questions  about  the  appraisal. 

For  government  source  selections,  this  could  be  an  official  bidders’ 
correspondence  procedure.  For  commercial  source  selections,  this 
could  be  email,  telecom,  briefings,  or  meetings  with  company 
representatives. 

In  either  case,  considerations  for  fair  competition  may  influence  the 
communication  strategy  as  well  as  the  strategy  for  obtaining  the  initial 
objective  evidence. 

Because  source  selection  requirements  may  include  constraints  on  the 
level  of  interaction  allowed  with  the  bidder,  initial  data  may  not  be 
available  for  team  training.  The  data  collection  plan  may  also  need  to 
be  updated  to  address  schedule  changes.  Updates  must  be  coordinated 
with  the  sponsor  to  determine  appropriate  actions,  which  might  include 
adjusting  the  appraisal  schedule  to  allow  for  a  discovery-mode 
appraisal. 
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1.5  Prepare  for  Collection  of  Objective  Evidence 


Overview 


Required  Practices: 
General 


Parameters  and  Limits: 
Data  Collection  Plan 


Parameters  and  Limits: 
Readiness  Review 


The  performance  of  a  readiness  review  provides  risk  mitigation  for 
situations  where  completion  of  the  appraisal  process  is  contingent  on 
sufficiency  of  data  examined  during  the  appraisal. 

As  a  more  general  technique,  great  utility  can  be  found  in  conducting 
readiness  reviews  to  assess  satisfaction  of  entry  criteria  for  any 
activity.  In  some  situations,  the  motivation  of  an  upcoming  readiness 
review  can  lead  to  clarification  of  vague  entry  criteria. 


The  appraisal  team  leader  shall 

•  Use  the  appraisal  input,  appraisal  plan,  and  other  artifacts  created 
in  planning  the  appraisal  to  plan  for  the  collection  of  objective 
evidence. 

•  Use  one  or  more  readiness  reviews  to  evaluate  the  feasibility  of  the 
plan  for  collecting  objective  evidence  and  the  plan  for  the  appraisal 
in  general. 

•  Make  minor  adjustments  or  major  revisions  to  the  plan  for 
collecting  objective  evidence,  as  needed. 


Appraisal  documents  created  during  planning  and  conduct  of  the 
appraisal  must  support  planning  and  replanning  the  collection  of 
objective  evidence.  An  artifact  named  “data  collection  plan”  is  not 
required,  but  is  a  recommended  best  practice  for  appraisals  with  large 
scopes.  Typically,  the  combination  of  the  on-site  schedule,  other 
elements  of  the  appraisal  plan,  and  the  data-tracking  tools  used  during 
the  appraisal  compose  (in  aggregate)  the  data  collection  plan. 


Information  resulting  from  each  readiness  review  must  be  reported  to 
the  sponsor  and  must  explicitly  address  the  feasibility  of  the  appraisal 
schedule. 


Con  tin  ued  on  next  page 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 
(continued) 


Parameters  and  Limits: 
Readiness  Review 
Pertaining  to  Objective 
Evidence 


In  some  applications,  the  ability  to  perform  a  robust  readiness  review 
may  be  limited  by  the  context.  For  example,  in  an  acquisition 
application,  the  accessibility  of  objective  evidence  may  be  limited  by 
the  classification  of  information  as  proprietary  or  classified.  In  such  a 
situation  at  least  one  readiness  review  must  still  be  performed.  In  fact, 
more  frequent  readiness  reviews,  if  feasible,  are  desirable  in  these 
situations  because  the  risks  associated  with  availability  of  objective 
evidence  may  be  more  severe. 


The  minimum  requirements  that  must  be  met  when  conducting  a 
readiness  review  focused  on  objective  evidence  are  as  follows: 

•  criteria  forjudging  the  sufficiency  of  data  are  established 

•  the  amount  and  type  of  data  required  is  specified 

•  actual  data  available  for  the  appraisal  are  inventoried 

•  results  of  the  review  are  documented 

•  results  of  the  review  are  communicated  to  the  sponsor 


Guidance:  Level  of 
Formality  in  the 
Readiness  Review 


The  focus  of  the  readiness  review  and  the  level  of  formality  warranted 
for  this  activity  should  be  tailored  to  match  the  context  of  the  appraisal. 
For  a  SCAMPI  C  conducted  by  a  single  appraiser  over  a  one-day 
period,  confirming  the  availability  of  documentation,  and/or  key 
interviewees  based  on  a  previous  agreement  may  be  all  that  is  needed. 
This  confirmation  could  be  accomplished  with  a  phone  call  or  an 
email.  In  contrast,  a  week-long  SCAMPI  B  using  a  team  of  appraisers 
may  require  a  two-hour  readiness  review  at  the  conclusion  of  team 
training,  where  a  set  of  artifacts  is  reviewed  for  relevance  to  the  model 
practices  and  the  availability  of  the  interviewees  is  confirmed. 


Guidance:  Continuous  As  the  team  collects  and  examines  data,  a  mechanism  should  be  used 
Consolidation  for  tracking  the  data  already  examined,  the  data  yet  to  be  examined, 

and  the  remaining  opportunities  to  collect  new  data.  Commercial 
software  tools  are  available  to  support  this  process. 

Planning  to  answer  the  following  three  questions  on  a  regular  basis 
allows  the  appraisal  activities  to  be  adjusted  as  needed: 

•  “What  data  do  we  have?” 

•  “What  data  do  we  need?” 

•  “Where  are  we  going  to  get  it?” 

For  a  multi-day  on-site,  at  the  conclusion  of  each  day,  a  review  of  the 
answers  to  the  above  questions  is  often  performed.  The  formality  of 
this  review  and  the  option  to  conduct  it  more  frequently  than  once  a 
day  is  a  matter  of  style  or  preference  on  the  part  of  the  appraisal  (team) 
leader. 


Continued  on  next  page 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 
(continued) 


Guidance  for  the  Conducting  a  thorough  readiness  review  is  important  in  determining 

Internal  Change  Agent  the  satisfaction  of  entry  criteria  for  conducting  the  appraisal.  The 
internal  change  agent  typically  resides  within  the  organization  and  may 
be  subject  to  more  pressures  to  proceed  in  the  absence  of  adequate 
preparation.  The  readiness  review  process  can  be  used  to  mitigate  this 
risk.  The  readiness  review  should  be  performed  far  enough  in  advance 
to  give  the  organization  time  to  collect  additional  evidence  to  support  a 
more  successful  appraisal. 

The  extent  of  the  readiness  review  will  vary  based  on  the  amount  and 
type  of  data  required.  For  example,  a  SCAMPI  C  could  be  limited  to 
interviews  only  or  use  of  instruments  only,  or  it  could  consist  of  a 
document  review  limited  to  examining  the  approach  (reviewing 
documented  policy  and  process  documentation)  versus  reviewing 
project  artifacts. 

The  focus  of  a  readiness  review  in  which  only  interviews  are 
conducted  would  include  a  check  of  the  availability  of  interviewees 
and  their  ability  to  address  information  needs.  A  readiness  review  of  an 
appraisal  limited  to  the  use  of  an  instrument  (survey)  would  include  a 
check  on  both  the  availability  and  appropriateness  of  the  individuals 
completing  the  survey.  The  focus  of  a  readiness  review  where  only  a 
document  review  is  conducted  would  include  a  check  of  the 
availability  of  the  appropriate  documentation  and  its  appropriateness  to 
address  information  needs. 

In  a  SCAMPI  C,  the  data  collection  plan  has  the  greatest  potential  for 
tailoring.  The  availability  of  direct  or  indirect  evidence  can  be 
augmented  with  interviews,  instruments,  or  presentations.  The  amount 
of  objective  evidence  collected  depends  on  the  appraisal  objectives. 
For  example,  if  the  appraisal  objective  is  to  expose  the  organization  to 
the  appraisal  process,  both  interviews  and  document  review  may  be 
warranted.  Determining  how  extensively  new  practices  are 
implemented  may  only  require  interviews  or  a  survey.  In  low-maturity 
organizations,  the  emphasis  is  often  on  the  organization’s  priorities, 
not  those  of  the  model,  which  allows  for  a  lot  of  leeway  in  the  amount 
and  type  of  objective  evidence  collected. 


Con  tin  ued  on  next  page 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 
(continued) 

Guidance  for  the  Timing  the  readiness  review  can  be  difficult  for  the  professional 

Professional  consultant,  who  often  has  a  very  limited  amount  of  time  on-site. 

Consultant  Allowing  for  corrective  actions  following  the  readiness  review  is  very 

difficult.  More  often,  the  professional  consultant  is  forced  to  react  to 
the  situation  by  adjusting  the  approach  when  the  satisfaction  of  entry 
criteria  is  not  feasible. 

As  described  earlier,  the  professional  consultant  will  often  arrange  for 
a  dedicated  resource  to  act  as  the  primary  point  of  contact.  This  person 
will  provide  essential  support  during  the  preparatory  activities.  This 
type  of  vehicle  for  gaining  access  to  information  is  essential  for  the 
professional  consultant,  who  is  typically  an  “outsider”  to  the 
organization. 

Guidance  for  the  The  external  auditor  is  sometimes  severely  constrained  in  the  amount 

External  Auditor  of  time  available  and  the  access  provided  to  gather  information  about 

the  organization  and  the  appraisal  participants.  More  formal 
communication  paths  are  frequently  necessary,  as  are  strategies  for 
pre-packaging  information  and  delegating  responsibilities  to  others. 

Readiness  reviews  may  be  performed  very  late  in  the  planning  process, 
even  on  the  first  day  of  data  collection,  if  appraisal  constraints 
necessitate.  If  the  readiness  review  reveals  significant  issues  with  the 
plan  for  the  appraisal,  then  formal  mechanisms  may  exist  to  require 
that  the  organization  to  be  appraised  first  correct  the  situation.  More 
likely,  the  appraisal  will  need  to  be  re -planned  or  reconfigured. 

Another  consideration  for  the  external  auditor  is  that  the  accessibility 
of  objective  evidence  may  be  limited  by  the  classification  of 
information  as  proprietary  or  classified.  Use  of  at  least  two  “cleared” 
team  members  is  strongly  recommended. 
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Phase  2 

Conduct  Appraisal 

Overview 

The  minimum  requirements  for  data  collection  processes  and  artifacts 
are  specified  in  this  section. 

2.1  Examine  Objective 
Evidence 

The  fundamental  process  of  collecting  relevant  data  about  the 
organizational  unit  and  relating  it  to  the  specific  and  generic  practices 
of  the  reference  model  is  similar  for  all  members  of  the  SCAMPI 
family  of  appraisals.  Minimum  standards  and  shared  operational 
definitions  provide  the  basis  for  upward  compatibility  of  the  data 
collected  on  all  SCAMPI  appraisals. 

2.2  Document 

Objective  Evidence 

ARC4.2 

Data  gathered  during  the  appraisal  must  be  consistently  recorded  and 
maintained  to  ensure  the  reliable  use  of  the  appraisal  methodology  and 
valid  appraisal  results.  The  minimal  standards  associated  with 
recording  data  provide  flexibility  in  implementation.  Non-attribution 
provisions  and  traceability  of  data  to  their  sources  is  maintained  at  all 
times. 

2.3  Verify  Objective 
Evidence 

ARC4.5.2 

Objective  evidence  is  examined  to  understand  the  practices  planned  or 
deployed  in  the  organization.  The  requirements  for  verification  of 
objective  evidence  address  the  use  of  direct  and  indirect  artifacts  of 
practice  implementation,  and  affirmations.  Activities  required  in  the 
processing  of  preliminary  findings  are  elaborated  as  well. 

2.4  Validate 

Preliminary  Appraisal 
Outputs 

Validation  of  preliminary  appraisal  results  is  a  hallmark  of  SCAMPI 
appraisals.  The  ideal  method  is  the  traditional  preliminary  findings 
presentation,  though  other  means  are  available.  The  SCAMPI  C 
method  allows  for  a  very  limited  validation  mechanism. 

2.5  Generate  Appraisal 
Results 

ARC4.6.1 

Variations  in  the  implementation  of  practices  across  parts  of  the 
organizational  unit  and/or  across  parts  of  the  appraisal  reference  model 
can  be  communicated  using  characterization  schemes.  Other  attributes 
of  the  data  can  be  characterized  to  aid  their  inteipretation  as  well.  Prose 
statements  relating  planned  or  enacted  practices  to  CMMI  are  standard 
output  of  SCAMPI. 
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2.1  Examine  Objective  Evidence 


Overview 


Required  Practices 

ARC.4.2.13 


Parameters  and  Limits: 
General 


This  section  addresses  fundamental  definitions  that  capture  the  intent 
of  data  collection  in  the  SCAMPI  B  and  C  methods. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 

used)  shall 

•  Seek  and  review  information  and  artifacts  that  relate  to  the 
approach,  deployment,  or  institutionalization  of  practices 

•  Annotate  appraisal  work  products  that  identify  information  or 
artifacts  reviewed 

•  Annotate  appraisal  work  products  to  indicate  the  portion  of  CMMI 
to  which  the  information  applies 

•  Annotate  appraisal  work  products  to  identify  the  portion  of  the 
organizational  unit  to  which  the  information  applies 


Oral  statements  and  written  information  examined  by  the  team  must 
serve  as  objective  evidence  of  the  approach,  deployment,  or 
institutionalization  of  practices  in  the  organization  that  support 
satisfaction  of  the  goals  found  in  CMMI.  (Note:  determining  ratings  of 
goal  satisfaction  is  permissible  only  in  a  SCAMPI  A  appraisal.) 

Sources  of  information — people,  documents,  or  other  work  products — 
must  be  recorded  in  order  to  support  bi-directional  traceability  between 
objective  evidence  and  the  appraisal  outputs  that  were  derived  from  the 
objective  evidence. 

The  relevance  of  the  information  examined  to  individual  practices  or 
other  components  of  CMMI  must  be  documented.  This  must  be  done 
in  a  way  that  allows  the  appraisal  (team)  leader  to  determine  the 
amount  of  objective  evidence  examined  that  relates  to  each  individual 
model  component  at  various  points  during  the  appraisal. 

The  association  of  information  examined  with  one  or  more  portions  of 
the  organizational  unit  must  also  be  documented.  This  must  be  done  in 
a  way  that  allows  the  appraisal  (team)  leader  to  determine  which 
information  applies  to  which  sampled  instantiation  (project  or  other 
work  group). 


Continued  on  next  page 
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2.1  Examine  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Objective  Evidence 

ARC.4.4 


Parameters  and  Limits: 
Practice 
Implementation 
Indicators 


The  term  “objective  evidence”  provides  a  basis  for  identifying 
information  that  supports  judgments  made  during  the  appraisal.  The 
SCAMPI  family  of  appraisals  defines  three  types  of  objective 
evidence: 

•  direct  artifacts 

•  indirect  artifacts 

•  affirmations 

Consistent  use  of  these  definitions  is  a  requirement  that  enables 
upward  compatibility  in  the  family  of  SCAMPI  methods. 


Practice  Implementation  Indicators  (PIIs)  provide  a  structure  for 
relating  individual  pieces  of  objective  evidence  to  particular  practices 
in  the  model,  as  well  as  to  particular  parts  of  an  organizational  unit. 
The  discussion  of  initial  objective  evidence  under  Process  1.4  provides 
additional  information  on  this  topic. 

PIIs  can  be  used  in  at  least  three  different  ways  in  an  appraisal,  to 
organize 

•  the  input  data  provided  to  the  team 

•  the  data  collected,  managed,  and  used  by  the  team 

•  the  output  of  the  appraisal 

As  a  data  structure,  the  use  of  PIIs  is  required.  Specific  requirements 
are 

•  the  classification  of  artifacts  into  direct  and  indirect  indicators  of 
practice  implementation 

•  labeling  of  affirmations  as  a  distinct  source  of  data 

•  associating  each  item  of  evidence  with  a  particular  practice  in  a 
CMMI  model  (or  a  non-model  category) 

•  associating  each  item  of  evidence  with  either  a  given  project  or  an 
organizational  function 

•  use  of  inventories  of  data  based  on  this  framework  to  establish  that 
sufficient  data  have  been  examined  to  support  appraisal  outcomes 

Use  of  this  data  structure  promotes  compatibility  across  the  SCAMPI 
family.  However,  there  is  no  requirement  to  structure  the  input  or 
output  this  way.  Deriving  “updated  PIIs”  as  an  output  to  a  SCAMPI  B 
is  strongly  recommended,  especially  in  settings  where  a  future  Class  A 
appraisal  is  anticipated. 


Con  tin  ued  on  next  page 


CMU/SEI-2005-HB-005 


2.1  Examine  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Work  Actually  Being 
Done 

ARC4.5.4.C 


Parameters  and  Limits: 
Interviews 

ARC.4.4.2 


Parameters  and  Limits: 
Remote  Interviews 


In  Class  B  methods,  the  ARC  requires  the  corroboration  of  data  that 
reflects  “work  actually  being  done  (e.g.,  process  area 
implementation).”  This  requirement  applies  to  verification  of  objective 
evidence  and  has  consequences  for  data  collection. 

This  requirement  ensures  that  data  collected  reflects  work  products  and 
results  from  the  implementation  of  one  or  more  practices.  This  is  in 
contrast  to  items  like  policies  and  implementation  guidance,  which 
support  consistent  use  of  practices. 

The  focus  on  deployment  in  the  SCAMPI  B  method  requires  that  direct 
artifacts  be  reviewed  in  order  to  ensure  coverage  of  “work  actually 
being  done.”  The  Class  C  method  (according  to  the  ARC)  does  not 
require  that  this  type  of  data  necessarily  be  included.  SCAMPI  C  can 
be  used  to  go  beyond  an  examination  of  “approach,”  in  which  case 
direct  artifacts  may  well  be  examined.  However,  there  is  no 
requirement  that  direct  artifacts  (or  “work  actually  being  done”)  be 
examined  in  a  SCAMPI  C. 


For  SCAMPI  B,  interviews  must  be  conducted  in  the  presence  of  at 
least  two  members  of  the  appraisal  team.  For  SCAMPI  C,  there  is  no 
requirement  to  use  a  team.  If  a  team  is  used  in  a  SCAMPI  C,  then 
interviews  may  be  conducted  by  individual  team  members. 


Because  of  the  increasing  prevalence  of  distributed  workforces  and 
strategies  that  leverage  the  concept  of  the  “virtual  enterprise,”  some 
appraisals  now  rely  on  technology  such  as  video  teleconferencing. 
SCAMPI  B  and  C  appraisals  allow  the  conduct  of  interviews  using 
such  technology. 

For  SCAMPI  B  and  C,  there  is  no  limitation  on  the  use  of  technology 
to  conduct  interviews.  Every  interview  can  be  conducted  using 
teleconference  or  video  teleconference  technology  if  interviews  are 
used  during  the  appraisal.  However,  for  SCAMPI  B,  it  is  not 
recommended  that  all  interviews  be  conducted  via  teleconference.  In 
cases  where  a  heavy  reliance  on  this  technology  is  anticipated,  it  is 
strongly  recommended  that  one  or  more  members  of  the  team  be 
present  at  the  remote  site.  This  arrangement  permits  at  least  one  team 
member  to  be  in  the  room  with  the  people  being  interviewed. 
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2.1  Examine  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Using  Presentations  as 
Interviews 


Parameters  and  Limits: 
Concurrent  Interviews 


The  use  of  standardized  presentations,  delivered  live  by  an 
“interviewee,”  is  another  variant  of  the  interview  process  permitted  in 
the  SCAMPI  B  and  C  methods. 

For  SCAMPI  B,  in  order  for  this  method  to  serve  the  same  role  as  an 
interview,  the  following  conditions  must  be  met: 

•  The  content  of  the  presentation  must  be  specified  in  advance,  with 
involvement  of  the  appraisal  (team)  leader. 

•  At  least  two  team  members  must  be  present  during  the 
presentation. 

•  Team  members  must  be  able  to  ask  clarifying  questions  and  seek 
related  information. 

•  The  person  making  the  presentation  must  be  a  manager  or 
practitioner  who  has  first-hand  experience  with  the  practices 
related  to  the  material  being  presented. 

There  are  no  limitations  on  the  use  of  presentations  for  SC  AMPI  C. 


When  using  an  appraisal  team,  it  is  conceivable  that  more  than  one 
interview  session  will  be  underway  simultaneously.  In  performing  a 
SCAMPI  C,  it  is  permissible  that  each  of  these  interviews  be 
conducted  by  a  single  team  member.  In  a  SCAMPI  B,  at  least  two  team 
members  must  be  present  in  each  interview  session.  Consideration 
must  be  given  to  achieving  consensus  when  performing  concurrent 
interviews. 
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2.1  Examine  Objective  Evidence  (continued) 


Guidance:  Instruments 

ARC.4.4.1 


Instruments  can  provide  a  relatively  low-cost  source  of  appraisal  data. 
This  form  of  data  collection  also  places  added  burden  on  the 
organization  to  supply  information  in  advance. 

Many  different  things  will  qualify  as  an  instrument  in  a  SCAMPI 
appraisal.  Instruments  may  be  data  tables,  templates,  forms, 
questionnaires,  surveys,  or  other  standardized  response  formats  for 
providing  data. 

When  instruments  are  self-administered  in  a  setting  with  little  past 
experience  or  support  to  aid  the  respondent,  they  can  yield  very  little 
useful  information  and  can  even  become  counterproductive.  Support 
for  explaining  the  intended  use  of  the  data  as  well  as  clarifications  for 
the  content  of  the  questions  are  usually  needed  to  ensure  that  data  are 
usable. 

Instruments  may  also  play  a  dominant  role  in  the  data  collection 
process,  as  when  Practice  Implementation  Indicator  databases  are  used 
to  organize  and  inventory  objective  evidence.  An  instrument-centric 
appraisal  method  employs  the  structure  and  content  of  a  questionnaire 
to  define  the  inventory  of  data  to  be  pursued  during  the  appraisal. 

The  information  summarized  in  an  instrument  is  not  always  combined 
directly  with  other  data  collected  during  an  appraisal.  For  surveys  and 
questionnaires,  a  summarization  or  tabulation  process  transforms  the 
data  before  they  are  used  to  support  creation  of  findings  and  other 
outputs  of  the  appraisal.  The  scheme  by  which  the  tabulations  and 
summaries  are  interpreted  will  be  defined  within  the  appraisal  outputs 
that  include  them. 
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2.1  Examine  Objective  Evidence  (continued) 


Guidance:  Interviews 


Interviews  are  viewed  as  a  very  flexible  and  useful  data  gathering 
activity.  The  ability  to  dynamically  change  topics  and  pursue 
additional  detail  as  opportunities  arise  makes  interviews  a  robust 
method  to  gather  data.  Successful  conduct  of  this  activity  is  also 
contingent  on  the  knowledge  and  skill  of  the  interviewer(s). 

At  the  most  fundamental  level,  an  interview  consists  of  a  verbal 
interaction  between  two  or  more  people,  from  which  data  related  to  the 
appraisal  are  obtained.  Many  different  types  of  interviews  are 
permitted  within  the  family  of  SCAMPI  appraisals.  The  major 
differentiators  among  different  types  of  interviews  are  summarized 
below. 

The  most  structured  approach  is  the  standard  interview ,  which  is 
scheduled  in  advance  and  notification  is  given  to  participants  that 
includes  detailed  expectations  for  topics  to  be  covered.  A  more  flexible 
approach  to  scheduling  interviews  is  available  in  the  form  of  on-call 
interviews.  Prospective  interviewees  are  identified  and  notified  in 
advance;  however,  the  interviews  are  only  held  as  needed.  Office-hour 
inter-views  represent  an  agreement  for  availability  that  permits  pairs  of 
team  members  to  visit  interviewees  at  their  offices.  The  prospective 
interviewees  block  off  a  specific  time  period  to  be  available  on  a 
contingency  basis. 

Interviews  may  also  take  on  a  collaborative  workshop  or 
brainstorming  approach,  with  relaxed  coverage  requirements  for 
objective  evidence.  In  this  approach,  interviewees  are  asked  to 
brainstorm  rather  than  provide  affirmations  about  objective  evidence. 
This  approach  involves  the  contribution  of  ideas  from  a  group  of 
interviewees  relative  to  broader  process  improvement  topics. 

Some  interviews  may  be  conducted  on  a  one-to-one  basis,  as  in 
SCAMPI  C  where  only  one  appraiser  might  be  used.  Other  interviews 
may  be  a  many-to-one  basis,  where  the  appraisal  team  (or  a  subset  of 
the  team)  interviews  a  single  person.  Still  other  interviews  may  be  a 
one-to-many  or  many-to-many  basis,  where  the  appraisal  team,  a 
subset  of  the  team,  or  a  single  appraiser  interviews  a  group  of  people. 
The  Parameters  and  Limits  section  covering  interviews  (above) 
clarifies  the  patterns  that  are  permissible  in  each  SCAMPI  method. 

Parallel  interviews  may  also  be  conducted.  This  allows  for  more  than 
one  interview  to  be  conducted  simultaneously  using  different  members 
of  the  team  as  interviewers.  In  planning  for  such  parallel  sessions,  care 
must  be  taken  to  assign  skilled  team  members  to  perform  interviews. 
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2.1  Examine  Objective  Evidence  (continued) 


Guidance:  Interviews 
(continued) 
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Interviews  may  also  rely  on  the  use  of  teleconferencing,  video 
teleconferencing,  or  virtual  meeting  technology  (e.g.,  Net  Meeting). 
Obviously,  there  are  implications  for  interviewee  comfort  as  well  as  a 
need  for  context  setting  that  may  be  increased  when  relying  on 
technology  in  this  way.  Also,  the  accuracy  and  precision  of  the 
appraisal  results  may  be  called  into  question  by  some  stakeholders  if 
they  perceive  an  over-reliance  on  teleconferences  as  the  primary  source 
of  data. 

The  Parameters  and  Limits  section  above  specifies  no  limitation  on 
who  can  be  interviewed  during  an  appraisal.  In  general,  people  will  be 
better  suited  to  serve  as  interviewees  for  questions  related  to  the  role 
they  play  in  the  organization.  Roles  typically  interviewed  include: 
senior  or  middle  management,  program  or  project  management, 
engineering  or  technical  staff,  support  or  administrative  staff,  and 
functional  area  specialists  (e.g.,  quality  assurance). 
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2.1  Examine  Objective  Evidence  (continued) 


Guidance:  Document 
Review 


Document  review  is  an  important  part  of  the  appraisal  process. 
Documents  come  in  many  forms  and  types  and  can  be  the  source  of 
information  for  a  wide  range  of  practices. 

Many  different  forms  of  information  may  be  treated  as  documents 
during  an  appraisal.  Most  broadly  defined,  any  lasting  or  reusable 
representation  of  relevant  information  can  be  considered  a  document 
for  the  purpose  of  an  appraisal. 

Documents  may  come  in  the  form  of  paper  copies,  electronic  files,  and 
media  stored  on  a  network,  including  database  queries,  computerized 
training  materials,  and  system-generated  records.  The  accessibility  of 
information  or  records  of  usage  (such  as  “hit  counts”  on  Web  pages) 
are  unique  considerations  of  the  acceptability  of  evidence  from 
document  reviews.  Such  things  are  not  typically  referred  to  as 
documents  in  other  contexts. 

Documents  can  be  classified  with  respect  to  the  level  of  the 
organizational  hierarchy  from  which  they  originate,  or  to  which  they 
apply.  There  is  often  a  meaningful  difference  between  organizational, 
division-  or  discipline-specific,  or  program-  or  project-specific 
documents.  Each  level  of  the  hierarchy  potentially  relates  to  different 
process  areas  and  practices. 

Some  documents  communicate  expectations  or  acceptable  tailoring 
limits  for  processes  in  the  organization/division/project.  Other 
documents  are  byproducts  of  implementing  the  process. 

For  example,  a  Capability  Level  1  implementation  is  typically  sought 
through  examining  operational  work  products,  while  Capability  Level 
2  and  above  will  often  involve  guidance  documents,  such  as  policies  or 
process  documentation  in  addition  to  the  operational  work  products. 
Depending  on  the  practice  under  consideration,  the  appropriate 
document  may  be  an  organizational  level  document,  a  project-specific 
work  product,  or  something  in  between. 

Document  review  conducted  during  an  appraisal  will  always  involve 
reading  and  understanding  the  content  of  the  document,  and  how  it 
pertains  to  the  practices  planned  or  implemented  in  the  organization.  A 
simple  inventory  of  documents  does  not  establish  objective  evidence 
for  an  appraisal.  The  role  played  by  the  document  in  the  process  is 
what  the  appraisal  process  examines. 


Con  tin  ued  on  next  page 


58 


CMU/SEI-2005-HB-005 


2.1  Examine  Objective  Evidence  (continued) 


Guidance: 

Presentations 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Presentations  can  be  used  during  the  conduct  of  appraisals  in  a  number 
of  different  ways.  The  focus  in  this  section  is  on  the  ways  that 
presentations  can  serve  as  a  data-collection  technique. 

Objective  evidence  may  be  presented  by  members  of  the  organization 
through  an  interactive  session  in  which  the  appraisal  team  can  receive 
explanations  by  someone  familiar  with  the  organization  and  can  ask 
specific  questions  about  the  information. 

Demonstrations  of  tools  or  products  may  include  presentation  material 
that  helps  to  orient  the  appraisal  team  members  as  they  examine  the 
role  of  the  tool  or  product  in  the  organization’s  process. 

Presentations  found  in  some  standard  project  status  reviews  can  be 
highly  informative  to  the  appraisal  process.  This  is  especially  true  for 
high-maturity  organizations  that  share  data  and  the  results  of  “Decision 
Analysis  and  Resolution”  with  affected  stakeholders. 


Given  the  contextual  knowledge  the  internal  change  agent  typically 
has,  he  or  she  can  often  be  sure  that  the  full  set  of  available 
documentation  has  been  reviewed.  However,  the  dynamics  of  the 
interview  process  may  be  more  challenging.  Existing  relationships  and 
the  need  to  promote  cooperation  for  ongoing  and  future  activities  will 
influence  the  interview  process. 

Interviewee  selection  may  include  a  mix  of  process  proponents  and 
naysayers.  Having  a  mix  of  interviewees  provides  the  appraisal  team 
with  a  better  understanding  of  the  organizational  culture.  This 
knowledge  may  help  determine  the  appropriate  approach  to  change  the 
culture. 

A  note  on  remote  interviews:  Reliance  on  technology  (e.g., 
teleconference,  video  teleconference,  Web  Ex)  to  support  interviews 
with  members  of  the  workforce  who  are  distributed  across  a  wide 
geographic  area  can  help  the  organization  meet  appraisal  cost 
constraints.  When  there  is  a  heavy  reliance  on  this  technology,  the  data 
collection  plan  may  be  adjusted  to  emphasize  the  use  of  direct  and  or 
indirect  evidence  to  augment  any  perceived  risks  relative  to  the 
accuracy  and  precision  of  the  appraisal  results. 


The  professional  consultant  must  build  rapport  with  members  of  the 
organization  in  order  to  ensure  that  sufficient  objective  evidence  can  be 
obtained  in  a  timely  manner.  Strategies  for  delegating  data-collection 
responsibilities  are  very  important  to  the  professional  consultant,  who 
does  not  typically  interact  daily  with  members  of  the  organization. 
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2.1  Examine  Objective  Evidence  (continued) 


Guidance  for  the 
External  Auditor 


The  external  auditor  may  not  always  be  viewed  as  a  welcome  visitor  to 
the  organization.  Interview  techniques  may  need  to  be  adjusted 
significantly  if  participants  are  not  comfortable  with  the  idea  of  frank 
and  open  dialog  about  the  way  they  do  their  jobs.  One  common 
approach  to  dealing  with  this  issue  is  to  rely  more  heavily  on 
presentations  offered  by  members  of  the  organization.  By  specifying 
the  required  content  of  the  presentation  in  advance,  specified  objective 
evidence  can  be  pursued  while  providing  the  presenter  (interviewee) 
with  an  opportunity  to  consider  their  responses  to  the  information 
requests. 


♦ 
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2.2  Document  Objective  Evidence 


Overview 


Required  Practices 


Parameters  and  Limits: 

Recording 

Affirmations 


Parameters  and  Limits: 
Recording  Deficiencies 


Parameters  and  Limits: 
Recording  Strengths 


Consistent  use  of  data-recording  procedures  ensures  that  key 
information  will  not  be  lost  and  that  the  data  will  be  treated 
appropriately  during  the  appraisal  process. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 

used)  shall 

•  Write  notes  and  annotate  worksheets  (as  appropriate)  to  capture 
information  that  serves  as  objective  evidence  or  supports  the  valid 
interpretation  of  objective  evidence 

•  Write  notes  and  annotate  worksheets  (as  appropriate)  to  document 
the  absence  of  objective  evidence  needed  to  verify  the  approach, 
deployment,  or  institutionalization  of  the  planned  or  deployed 
practices  in  the  organizational  unit 

•  Document  issues  or  gaps  in  the  approach,  deployment,  or 
institutionalization  of  practices  that  support  achievement  of  CMMI 
goals  in  the  process  context  at  hand 

•  Periodically  compare,  combine,  and  consolidate  documented 
information,  performing  a  reconciliation  of  differences  that  might 
arise  in  the  understanding  of  the  objective  evidence 

•  Periodically  inventory  the  objective  evidence  reviewed  to  identify 
the  information  already  collected,  the  information  yet  to  be 
collected,  and  the  set  of  remaining  activities  available  to  gather 
that  information 


Affirmations  provided  during  interviews  must  be  written  in  notes  or 
identified  on  worksheets  in  order  to  be  treated  as  objective  evidence. 
Recollections  of  the  interviewer(s)  are  not  adequate  to  support  the 
characterization  of  practices  or  to  derive  findings  and  other  appraisal 
outputs. 


Where  objective  evidence  is  judged  to  be  missing  or  deficient  in  some 
way,  a  written  statement  or  worksheet  annotation  that  will  be  used  in 
deriving  appraisal  outputs  must  be  created. 


Approaches  or  implementations  that  represent  exemplary  practices  (in 
reference  to  CMMI  goals)  should  be  documented  in  the  worksheet  and 
appraisal  outputs  as  strengths.  Practices  that  meet  the  intent  of  the 
model  are  not  necessarily  to  be  documented  as  strengths  unless  the 
method  is  tailored  to  provide  this  additional  output. 
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2.2  Document  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Traceability 

ARC4.2.16  (1) 


Parameters  and  Limits: 
Non-Attribution 

ARC4.2.15 


Guidance:  Continuous 
Consolidation 


Guidance: 
Tabular  Data 


As  individual  elements  of  data  are  combined,  the  origin  of  the  data 
being  combined  must  be  recorded.  The  sphere  of  applicability  for  the 
combined  data  must  be  understood.  Maintaining  the  traceability  back 
to  data  sources  also  supports  the  verification  processes  described  later 
in  this  handbook. 


Protecting  the  anonymity  of  individuals  who  provide  data  is  a 
cornerstone  principle  of  process  appraisals.  Non-attribution  and/or 
confidentiality  agreements  are  established  during  planning  and 
communicated  when  participants  are  prepared  for  the  appraisal. 

The  interim  work  products  of  the  appraisal  process  must  not  be  made 
available  to  others;  this  helps  to  protect  the  identity  of  people  who 
supply  information.  In  generating  appraisal  outputs,  data-aggregation 
strategies  are  used  to  generalize  beyond  identifiable  individuals. 
Specific  strategies  that  provide  sub-groupings  of  data  must  be 
negotiated  with  the  sponsor  and  consistently  communicated  to  all 
appraisal  participants  before  they  provide  data  to  the  appraisal. 


In  order  to  effectively  manage  the  appraisal  process,  one  must  use 
effective  methods  for  maintaining  an  inventory  of  data.  Effective 
methods  are  those  that  help  to  answer  the  following  three  questions: 

•  What  data  do  we  have? 

•  What  data  do  we  need? 

•  Where  am  I  going  to  get  the  needed  data? 

Integrated  software  tools  are  often  used  to  support  this  process.  Such 
tools  typically  permit  a  user  to  summarize  data  that  answer  the  first 
two  questions.  When  the  data-collection  plan  is  integrated  with  the 
data-tracking  tool,  the  user  is  also  able  to  “allocate”  data  needs  to  data- 
collection  events.  The  data-collection  plan  is  updated  as  required 
during  the  appraisal  as  objective  evidence  is  found  or  new  sources  of 
information  are  uncovered. 


Frequently,  tabular  summaries  of  instrument  data  (including 
questionnaires  as  well  as  Practice  Implementation  Indicator 
descriptions)  communicate  significant  patterns  in  the  appraisal  data. 
When  such  patterns  in  the  raw  data  are  used  in  the  context  of  appraisal 
outputs,  they  are  typically  accompanied  by  written  statements  that 
summarize  the  pattern  and  its  significance. 
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2.2  Document  Objective  Evidence  (continued) 


Guidance:  Practice 
Level  Data 


Guidance: 

Organizational  Level 
Findings 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 
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The  minimum  level  of  abstraction  expected  in  the  data  being  combined 
and  consolidated  is  a  focus  at  the  practice  level.  Findings  frequently 
span  multiple  practices  (or  even  cross  process  area  boundaries),  so  a 
one-to-many  mapping  between  findings  and  practices  is  not 
uncommon.  Flowever,  the  opposite  is  not  very  likely.  For  example, 
large  numbers  of  findings  covering  detailed  subpractices  are  not 
expected. 

There  are  times  when  a  more  detailed  focus  is  appropriate.  For 
example,  in  an  appraisal  that  serves  as  a  readiness  review  for  a 
benchmarking  appraisal,  a  focus  on  attributes  of  individual  artifacts 
may  be  appropriate. 


Data  collected  during  an  appraisal  are  typically  associated  with  a 
particular  instance  of  the  practice  implementation  in  the  organization. 
Commonly  this  is  understood  in  terms  of  the  implementation  of  a 
given  practice  in  a  specified  project.  With  CMMI,  implementations  in 
different  divisions  (or  discipline  groups)  may  also  come  into  play  when 
considering  the  aggregation  of  data. 

Findings  are  worded  (or  constructed)  to  focus  on  the  organizational 
unit  that  is  the  subject  of  the  appraisal.  Tailoring  options  to  generate 
appraisal  outputs  at  a  lower  level  of  aggregation  are  available. 
Appraisal  participants  are  informed  of  the  level  of  aggregation  used  in 
appraisal  outputs  if  information  will  be  reported  below  the  level  of  the 
organizational  unit. 


The  internal  change  agent  typically  has  knowledge  and  experience  that 
aids  in  the  careful  wording  of  findings.  Balancing  the  “word-smithing” 
issues  with  the  substantive  issues  may  be  difficult  at  times.  Flowever, 
people  who  are  very  familiar  with  an  organization  may  be  able  to 
choose  words  that  communicate  difficult  issues  most  clearly.  Such 
people  may  also  have  inhibitions  that  limit  their  ability  to  see  the  root 
cause  of  issues  that  are  very  close  to  them. 


Professional  consultants  must  be  quick  studies  to  ensure  that  their 
communications  and  the  way  that  they  characterize  the  practices  of  the 
organization  are  well  received.  Learning  to  “speak  the  organization’s 
language”  is  important.  This  is  one  of  the  areas  where  effective 
consultants  differentiate  themselves  from  others. 
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2.2  Document  Objective  Evidence  (continued) 


Guidance  for  the 
External  Auditor 


Taking  great  care  in  maintaining  traceability  is  a  higher  priority  for  the 
external  auditor  than  for  others  who  lead  appraisals  in  an  organization. 
The  defensibility  of  the  appraisal  outputs  may  not  rest  on  the 
sentiments  of  the  appraisal  participants,  who  may  never  interact  with 
the  sponsor  (who  is  external  to  the  organization).  Clear  traceability  of 
objective  evidence  and  data  sources  serves  the  external  auditor  well. 


♦ 
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2.3  Verify  Objective  Evidence 


Overview 

ARC4.5.3 


Required  Practices 

ARC.4.5.1 


Parameters  and  Limits: 
General 


Parameters  and  Limits: 
Verify 

Interdependencies 
Among  Objective 
Evidence 


The  data  used  to  formulate  appraisal  outputs  must  be  verified  to  ensure 
that  the  results  of  aggregating  individual  detailed  data  items  will  lead 
to  appropriate  appraisal  outputs. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 

used)  shall 

•  Verify  that  direct  artifacts,  indirect  artifacts,  affirmations,  and 
other  information  relating  to  each  model  component  meet  the 
appropriate  criteria  specified  in  Parameters  and  Limits  below 

•  Verify  that  criteria  for  data  sufficiency  are  satisfied  and,  if  a  team 
is  used,  that  this  verification  is  done  with  the  team’s  consensus 

•  Verify  that  each  preliminary  finding  is  supported  by  objective 
evidence  that  meet  the  specified  criteria 


For  SCAMPI  B,  objective  evidence  must  be  sought  for  each  individual 
practice  and  instantiation  included  in  the  scope  of  the  appraisal.  For 
each  practice,  at  least  one  direct  artifact  or  one  oral  affirmation  must  be 
obtained  for  each  instantiation.  In  addition,  for  each  practice  within  the 
scope  of  the  appraisal,  at  least  one  oral  affirmation  and  one  direct 
artifact  must  be  present  when  considering  the  set  of  instantiations  for 
that  practice. 

In  the  case  of  a  SCAMPI  B  focused  on  a  single  instantiation,  a  direct 
artifact  must  be  obtained  for  every  practice.  In  addition,  at  least  one 
oral  affirmation  must  be  obtained  for  at  least  one  practice  from  the  set 
of  practices  mapped  to  a  goal. 

For  SCAMPI  C,  at  least  one  item  of  objective  evidence  must  be  sought 
for  each  model  component  included  in  the  scope  of  the  appraisal. 
Whether  the  model  scope  is  defined  in  terms  of  individual  practices, 
goals,  process  areas,  or  other  model  component,  at  least  one  item  of 
objective  evidence  is  required  for  each  sampled  element  (practice, 
goal,  process  area,  or  other  model  component).  Direct  artifacts,  indirect 
artifacts,  oral  affirmations,  and  other  affirmations  (e.g.,  survey  data) 
may  be  used,  as  determined  by  the  judgment  of  the  SCAMPI  team 
leader. 


Items  of  objective  evidence  that  relate  to  one  another  must  be  reviewed 
to  verify  that  interrelationship.  For  example,  in  cases  where  standards 
and  policies  mandate  the  creation  of  an  artifact  that  meets  a  given 
standard,  the  existence  and  form  of  that  artifact  must  be  verified. 
Interpretations  of  the  set  of  objective  evidence  must  support  internally 
self-consistent  conclusions. 
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2.3  Verify  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Verify  Accuracy 

ARC4.5.2.a 


Parameters  and  Limits: 
Verify  Data  Sources 


Parameters  and  Limits: 
Verify  Non-Attribution 

ARC.4.2.15 

ARC4.5.2.b 


Parameters  and  Limits: 
Verify  Model 
Relevance 

ARC.4.2.16(1) 

ARC.4.5.1 

ARC4.5.2.C 


Parameters  and  Limits: 
Verify  Consistency 

ARC4.5.3.b 


Preliminary  findings  must  be  accurate,  in  that  they  are  derived  from 
what  is  seen  or  heard  during  data  collection,  worded  clearly  using 
terminology  that  is  understood  by  the  intended  audience,  and 
associated  with  a  particular  model  component. 


The  preliminary  findings  must  be  identified  with  the  sources  of  data 
from  which  they  were  derived.  This  identification  must  include  the 
data-gathering  event  (time/place)  and  the  information  provider  or 
location.  This  identification  is  not  to  be  made  visible  to  members  of  the 
appraised  organization.  Wording  of  preliminary  findings  must  not 
contain  generalizations  that  exceed  the  logical  scope  suggested  by  the 
identified  data  sources. 


The  preliminary  findings  must  be  clearly  worded,  without  attribution 
of  information  to  the  individuals  who  provided  it.  Subtle  wording 
choice  may  have  unique  significance  within  a  given  organization.  The 
history  of  the  organization  and  the  memorable  events  of  the  past  often 
lead  to  the  identification  of  certain  “loaded  words.”  Such  words  and 
associated  phrases  must  be  avoided  in  the  wording  of  preliminary 
findings. 


Preliminary  findings  related  to  the  model  are  expected  to  comprise  the 
majority  (if  not  the  totality)  of  the  appraisal  focus.  Including  “non¬ 
model”  findings  is  permitted. 

Model-related  preliminary  findings  must  be  accurately  traced  to  the 
model  components  included  in  their  scope.  It  is  expected  that  at  least 
one  practice  in  the  model  scope  of  the  appraisal  will  be  identified,  and 
that  this  relationship  will  be  verified.  Where  consensus  decision 
making  is  used  (it  is  required  for  SCAMPI  B)  the  team  must  achieve 
consensus  on  the  mapping  of  preliminary  findings  to  particular  model 
components. 


The  preliminary  findings,  taken  as  a  set,  must  be  internally  consistent. 
Apparent  (or  actual)  contradictions  among  or  within  preliminary 
findings  that  arise  from  unfortunate  wording  choices  must  be 
discovered  and  eliminated. 
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2.3  Verify  Objective  Evidence  (continued) 


Parameters  and  Limits: 
Verify  Corroboration 

ARC4.5.3.a 

ARC4.5.4.a-b 


Parameters  and  Limits: 
Work  Actually  Being 
Done 

ARC4.5.4.C 


Guidance:  “Discovery” 
vs.  “Verification” 


The  SCAMPI  B  method  requires  that  preliminary  findings  be 
corroborated  according  to  the  following  criteria: 

•  they  are  based  on  data  from  at  least  two  different  sources  (i.e., 
originating  from  two  different  individuals) 

•  they  are  based  on  data  from  at  least  two  data-gathering  sessions 

•  at  least  one  source  must  derive  from  work  actually  being  done* 

•  The  phrase  “work  actually  being  done”  is  elaborated  below. 


In  the  context  of  an  appraisal,  the  phrase  “work  actually  being  done” 
helps  to  define  a  class  of  data  that  is  expected  to  provide  more  direct 
and  credible  objective  evidence  than  data  from  other  sources. 

With  respect  to  data  from  instruments.  Practice  Implementation 
Indicator  descriptions  contain  direct  artifacts  that  result  from  execution 
of  the  practice  under  consideration. 

In  interview  data,  descriptions  of  actual  events  by  actors  in  those 
events  are  more  direct  evidence  of  the  practice. 

When  performing  document  reviews,  artifacts  that  derive  from 
operational  work  products  are  more  directly  indicative  of  practice 
implementation.  For  practices  implemented  at  the  project  level, 
project-level  work  products  are  more  likely  to  be  relevant  than 
command  media  such  as  policies  and  process  descriptions.  For 
practices  implemented  across  a  larger  group  in  the  organization, 
command  media  such  as  policies  and  process  descriptions  may 
represent  operational  work  products. 


The  distinction  between  “discovery”  and  “verification”  appraisals  is 
quite  important  in  the  context  of  a  SCAMPI  A.  In  some  applications  of 
SCAMPI  B,  a  verification  approach  will  be  highly  desirable,  especially 
if  the  appraisal  is  being  conducted  as  a  “dress-rehearsal”  for  a  future 
SCAMPI  A.  In  contrast,  it  may  not  be  feasible  to  use  a  verification 
approach  in  conducting  a  SCAMPI  C,  especially  if  the  appraisal  is 
being  conducted  as  a  “getting  started  intervention,”  where  the  objective 
is  to  educate  staff  on  the  model  and  process  improvement,  while 
identifying  the  critical  issues  that  motivate  people  to  participate  in  the 
process  improvement  program. 
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2.3  Verify  Objective  Evidence  (continued) 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Guidance  for  the 
External  Auditor 


Use  of  appraisals  in  the  context  of  an  ongoing  program  of  model-based 
improvement  is  typically  motivated  by  the  need  to  frequently  know  the 
status  of  the  improvement  activities  and  how  they  contribute  to 
meeting  the  organization’s  goals.  Tailoring  the  level  of  rigor  applied  in 
the  appraisal  process  can  facilitate  efficient  data  collection  for  this 
purpose.  In  monitoring  progress,  the  appraisal  may  focus  more  rigor  on 
“new  process  areas”  and  spend  relatively  less  time  investigating 
process  areas  that  have  previously  been  judged  “satisfied”  in  a 
benchmarking  appraisal.  This  sort  of  hybrid  approach  would  entail 
applying  the  standards  of  the  SCAMPI  B  to  some  areas,  and  the 
standards  of  SCAMPI  C  to  others.  Conversely,  in  performing  a 
SCAMPI  A,  corroboration  rules  must  be  adhered  to  for  all  process 
areas  regardless  of  whether  the  process  areas  were  satisfied  in  a  prior 
SCAMPI  B  or  SCAMPI  C. 


With  diverse  experience  working  in  a  variety  of  organizational 
settings,  the  professional  consultant  often  has  well-established  ways  of 
communicating  the  methods  for  achieving  the  verification  activities 
described  here.  Sometimes  a  data-management  tool  is  used  to  embody 
these  practices.  Clear  and  effective  training  on  the  use  of  the  tool  is 
necessary  for  the  team  to  get  the  most  benefit  from  their  use. 


As  when  objective  evidence  is  verified,  the  external  auditor  has  a  need 
for  clear  and  defensible  traceability  of  data.  Though  this  traceability 
may  be  withheld  from  reports  to  the  members  of  the  organization,  the 
external  sponsor  will  need  assurance  that  the  appraisal  outputs  are  well 
founded. 
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2.4  Validate  Preliminary  Appraisal  Outputs 


Overview 


Required  Practices 

ARC.4.5.7 


Parameters  and  Limits: 
General 


Parameters  and  Limits: 
Preliminary  Findings 
Presentation 


Use  of  a  “preliminary  findings  presentation”  is  a  longstanding  value- 
added  component  of  process  appraisals.  Other  ways  of  validating 
preliminary  results  are  also  available,  but  experience  shows  that  the 
traditional  preliminary  findings  presentation  is  a  very  powerful  tool. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 
used)  shall 

•  Assure  that  preliminary  versions  of  key  appraisal  outputs  are 
validated  by  the  set  of  stakeholders  defined  in  the  appraisal  plan  (if 
any  are  defined) 

•  C  onsider  feedback  obtained  during  validation  of  preliminary 
appraisal  outputs  in  revising  the  outputs 


When  performing  a  SCAMPI  B,  preliminary  findings  statements  must 
be  validated  with  appraisal  participants  who  provided  data.  Not  every 
appraisal  participant  must  be  present,  though  that  is  recommended.  At 
a  minimum,  there  must  be  a  representative  from  each  project  and 
functional  group  that  provided  data  for  the  appraisal. 

When  performing  a  SCAMPI  C,  validation  with  appraisal  participants 
is  strongly  recommended  but  not  required.  Stakeholders  who  review 
the  preliminary  results  of  a  SCAMPI  C  may  be  limited  to  appraisal 
team  members  and/or  the  sponsor. 

Statements  of  weaknesses  must  be  validated.  Validation  of  other 
preliminary  appraisal  outputs  (e.g.,  strengths  or  characterizations)  is 
recommended  but  not  required. 


When  presenting  preliminary  findings  for  validation,  justification  of 
the  findings  statements  must  be  avoided.  Elaborations  that  clarify  the 
intent  of  the  findings  statements  may  be  offered,  if  the  participants  in 
the  validation  session  have  questions. 

The  presentation  is  typically  delivered  by  the  appraisal  (team)  leader. 
However,  members  of  the  team  (if  a  team  is  used)  may  serve  as 
presenters,  at  the  sole  discretion  of  the  team  leader. 

The  presenter  must  accept  feedback  from  the  audience  and  avoid 
making  promises  to  change  the  content  of  the  findings  based  on 
comments  made  during  the  session. 

Confidentiality  of  data  sources  must  be  maintained  during  this  and  any 
other  activity  in  the  appraisal  process.  This  extends  to  the  participants 
in  the  preliminary  findings  briefing  as  well.  The  presenter  must  make  it 
clear  to  the  audience  that  they  are  not  to  take  notes,  or  repeat  what  is 
heard  during  the  presentation. 
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2.4  Validate  Preliminary  Appraisal  Outputs  (continued) 


Guidance:  Validation 
Techniques 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Guidance  for  the 
External  Auditor 


The  preliminary  findings  presentation  is  the  most  common  method  of 
performing  validation.  However,  alternative  ways  of  obtaining 
feedback — such  as  the  use  of  an  instrument  listing  each  preliminary 
finding — may  be  employed.  Focus  groups  that  provide  a  forum  for 
discussing  a  subset  of  the  preliminary  findings  may  also  be  useful. 


All  presentation  of  information  relating  to  process  improvement 
actions  can  have  a  consistent  theme  in  an  organization — one  that 
reinforces  the  motivations  of  the  appraisal  sponsor.  The  internal 
change  agent  is  typically  responsible  for  communicating  a  coherent 
picture  of  the  process  improvement  program,  and  this  will  shape  the 
content  and  style  of  the  preliminary  findings  presentation. 


Validation  of  preliminary  findings  with  the  appraisal  participants  is  a 
very  powerful  tool.  Having  an  opportunity  to  provide  feedback  on 
appraisal  findings  in  this  setting  is  important  because  the  professional 
consultant  may  be  able  to  create  a  safe  environment.  As  an  outsider, 
the  consultant  may  be  able  to  communicate  information  as  a  neutral 
party  who  is  not  viewed  as  taking  one  side  or  another  on  contentious 
issues. 

However,  when  not  practical,  validations  of  preliminary  findings 
should  occur  with  defined  stakeholders.  Stakeholders  must  include  the 
sponsor.  Typical  stakeholders  include  representation  from  the  process 
group,  key  project  managers,  and  individuals  with  change  authority. 


Constraints  deriving  from  an  acquisition  or  contractual  process  may 
preclude  validating  findings  with  appraisal  participants  using  the 
traditional  presentation.  An  approach  to  validating  findings  should  be 
established  early.  There  may  be  ways  of  using  a  formal  communication 
process.  Another  option  is  to  have  follow-up  focus  group  interviews 
with  questions  that  target  areas  of  weakness. 
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2.5  Generate  Appraisal  Results 


Overview 


Required  Practices 


Parameters  and  Limits 

ARC.4.2.14 

ARC.4.5.1 


Parameters  and  Limits: 
Findings 

ARC.4.2.14 


Parameters  and  Limits: 
Detailed  Model  Scope 


Appraisal  results  in  the  form  of  findings  (statements  of  strengths  and 
weaknesses)  are  intended  to  support  the  needs  of  appraisal  sponsors. 
Characterization  scales  are  also  frequently  employed  to  help  illustrate 
trends  and  patterns  in  appraisal  data. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 
used)  shall 

•  Document  statements  of  strengths,  weaknesses,  or  other  written 
expressions  of  appraisal  outcome 

•  Document  detailed  model  scope  of  the  appraisal  to  be  reported 

•  Document  characterizations  (as  appropriate)  for  each  model 
component  in  the  scope  of  the  appraisal 


A  set  of  descriptive  or  comparative  statements  must  be  generated  as  a 
result  of  every  appraisal.  These  statements  typically  take  the  form  of 
strengths  or  weaknesses  in  the  intended  or  deployed  practices.  Such 
statements  are  referred  to  generically  as  findings.  Findings  may  take 
many  forms  in  SCAMPI  B  and  C,  including 

•  statements  of  weaknesses  or  inadequacies  found 

•  explanations  of  strengths  or  positive  attributes  found 

•  identification  of  best  practices  and  where  they  reside 

•  summary  statements  that  accompany  data  profiles 

Characterizations  are  an  optional  output  of  SCAMPI  C,  but  a  required 
output  of  SCAMPI  B. 

Whenever  an  appraisal  team  is  used,  all  appraisal  outputs  must  be 
derived  through  team  consensus. 


Based  on  issues,  or  gaps  identified  (see  2.2  Document  Objective 
Evidence),  preliminary  findings  must  be  documented.  These 
preliminary  findings  are  then  verified  (2.3)  and  validated  (2.4)  before 
they  become  appraisal  outputs. 


Because  the  SCAMPI  B  and  SCAMPI  C  methods  permit  great 
flexibility  in  determining  the  model  scope,  a  simple  list  of  process  area 
names  will  not  always  be  sufficient  to  document  the  appraisal  scope. 
When  individual  practices  or  other  groups  of  model  content  are 
selected  that  don’t  correspond  to  sets  of  entire  process  areas,  a  more 
detailed  description  is  needed.  A  list  of  individual  practices  or  a 
graphical  representation  that  communicates  the  list  must  be  used  in 
cases  where  the  model  scope  cannot  be  accurately  conveyed  with  a  list 
of  process  areas. 
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2.5  Generate  Appraisal  Results  (continued) 


Parameters  and  Limits: 
SCAMPI  C 
Characterization 


If  characterizations  are  to  be  generated  during  the  conduct  of  a 
SCAMPI  C,  the  following  conditions  must  be  met: 

•  The  characterization  scale  used  must  be  a  three-point  scale. 

•  The  three-point  scale  may  include  additional  values  for  “not  yet” 
as  well  as  “out  of  scope,”  as  illustrated  in  the  guidance  below. 

•  Every  characterization  generated  must  be  supported  by  relevant 
objective  evidence. 

•  The  characterization  scale  must  not  overstate  the  conclusiveness  of 
the  outcome  (e.g.,  referencing  satisfaction  of  related  goals,  process 
areas,  or  capability  or  maturity  levels). 

Characterization  schemes  used  in  SCAMPI  C  must  be  explained  in  the 
appraisal  plan,  along  with  applicable  data-sufficiency  rules.  These 
data-sufficiency  rules  must  be  based  on  the  level  of  granularity 
intended  in  the  data  collection  (e.g.,  each  practice,  each  goal,  each 
process  area). 


In  SCAMPI  C,  data  may  be  examined  at  a  higher  level  of  granularity 
than  an  individual  practice.  It  is  possible  that  no  individual  practice  is 
characterized  because  all  data  are  examined  at  a  higher  level  of 
abstraction — for  example,  at  the  goal,  process  area,  maturity  level,  or 
process  area  category  level. 
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2.5  Generate  Appraisal  Results  (continued) 


Parameters  and  Limits: 
Practice 

Characterization  for 
SCAMPI  B 


For  SCAMPI  B,  every  practice  (specific  and  generic)  within  the  scope 
of  the  appraisal  must  be  characterized  for  each  instantiation  within  the 
appraisal  scope  through  a  consensus  process  using  the  scale  described 
in  the  table  below. 


Label 

Meaning 

Red 

The  intent  of  the  model  practice  is  judged  to  be  absent  or 
poorly  addressed  in  the  set  of  implemented  practices;  gaps 
or  issues  that  will  prevent  goal  achievement,  if  the 
deployment  occurred  in  this  way  across  the  organizational 
unit,  were  identified. 

Yellow 

The  intent  of  the  model  practice  is  judged  to  be  partially 
addressed  in  the  set  of  implemented  practices;  some  gaps 
or  issues  were  identified,  which  might  threaten  goal 
achievement  if  the  deployment  occurred  in  this  way  across 
the  organizational  unit. 

Green 

The  intent  of  the  model  practice  is  judged  to  be  adequately 
addressed  in  the  implemented  set  of  practices  examined  in 
a  manner  that  would  support  goal  achievement,  if  the 
practice  were  deployed  across  the  organizational  unit. 

In  addition  to  the  above,  a  designation  of  “out  of  scope”  is  used  when 
no  characterization  was  assigned  because  the  appraisal  did  not  gather 
data  to  support  characterization  of  the  practice.  Also,  a  designation  of 
“not  yet”  may  be  used  to  indicate  that  there  was  no  opportunity  to 
observe  the  implementation  of  the  practice  due  to  the  fact  that  none  of 
the  sampled  parts  of  the  organization  have  reached  the  phase  in  the  life 
cycle  where  the  practice  would  be  implemented. 


Parameters  and  Limits: 
Instance  and 
Organizational  Unit 
Characterizations 


For  SCAMPI  B,  characterizations  must  be  performed  at  the  instance 
level.  Characterization  at  the  organizational  unit  (OU)  level  is  not 
required;  but  if  OU  characterizations  are  generated,  they  must  be  based 
on  an  aggregation  of  the  instance  level  characterizations,  using  the 
aggregation  rules  specified  below. 


For  SCAMPI  C,  characterizations  may  be  performed  at  the  instance 
level  or  at  the  OU  level.  If  OU  characterizations  are  generated,  they 
may  be  based  on  aggregation  of  instance  characterizations,  or  they  may 
be  based  on  data  summarized  at  the  OU  level. 
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2.5  Generate  Appraisal  Results  (continued) 


Parameters  and 
Limits:  Aggregating 
Instance  Level 
Characterizations 


Guidance:  Practice 
Characterization  in 
SCAMPI  C 


In  generating  OU  characterizations  for  each  practice  in  SCAMPI  B,  the 
following  aggregation  scheme  is  required.  _ 


Instance  Characterizations 

Resulting  OU 
Characterization 

All  instantiations  characterized  Red 

Red 

All  instantiations  characterized  Yellow 

Yellow 

All  instantiations  characterized  Green 

Green 

All  conditions  not  included  above 

T earn  judgment — subj  ect 
to  below* 

*  The  OU  characterization  shall  not  be  Red  unless  at  least  one  instance 
characterization  is  Red,  and  the  OU  characterization  shall  not  be  green 
unless  at  least  one  instance  characterization  is  green 

Appraisal  teams  use  professional  judgment,  in  light  of  the  needs  of  the 
organization,  to  determine  OU  characterizations  for  practices  where  not 
all  instantiations  are  Red  and  where  not  all  instantiations  are  Green. 
When  the  team  elects  to  set  rules  of  thumb  (e.g.,  if  at  least  one  instance 
is  red,  then  the  OU  is  red),  these  rules  must  be  documented  in  the 
appraisal  plan  and  reported  as  part  of  the  appraisal  results. 


The  scheme  below  is  one  example  of  a  characterization  scale  that  can 
be  used  for  SCAMPI  C. 


Label 

Meaning 

Low 

The  intent  of  the  model  practice  is  judged  absent  or 
inadequately  addressed  in  the  approach;  goal  achievement 
is  judged  unlikely  because  of  this  absence  or  inadequacy. 

Medium 

The  intent  of  the  model  practice  is  judged  to  be  partially 
addressed  in  the  approach,  and  only  limited  support  for 
goal  achievement  is  evident. 

High 

The  intent  of  the  model  practice  is  judged  to  be  adequately 
addressed  in  the  set  of  practices  (planned  or  deployed)  in  a 
manner  that  supports  achievement  of  the  goal  in  the  given 
process  context. 

In  addition  to  the  above,  a  designation  of  “out  of  scope”  is  used  when 
no  characterization  was  assigned  because  the  appraisal  did  not  gather 
data  to  support  characterization  of  the  practice. 


Note  that  in  this  case,  because  of  the  focus  on  “approach,”  the 
designation  of  “not  yet”  may  not  apply. 
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2.5  Generate  Appraisal  Results  (continued) 


Guidance:  Practice 

Characterization 

Profile 


Guidance:  Findings 
Organized  by  Model 
Content  or 

Organizational  Scope 


Guidance: 

Data  Sufficiency 
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A  profile  of  characterizations  relating  to  individual  practices  in  the 
model  (and  across  elements  of  the  organization)  is  often  used  to 
communicate  status  of  improvement  relative  to  the  model.  Such 
profiles  can  communicate  a  summary  (broad)  view,  while  at  the  same 
time  identifying  detailed  (narrow)  information  that  stands  out  from 
other  information. 


Most  appraisal  findings  are  organized  in  accordance  with  content  of 
the  model.  However,  there  are  situations  where  summarizing  across  a 
variety  of  model  structures  might  be  advantageous  (e.g.,  when  there  are 
common  themes  associated  with  generic  practices,  in  addition  to 
themes  by  process  area). 

In  some  settings,  findings  associated  with  particular  projects  or  units 
within  the  scope  of  the  appraisal  are  desired.  Obviously  the  wisdom  of 
providing  such  potentially  targeted  results  must  be  considered  during 
the  process  of  establishing  the  appraisal  input. 


SCAMPI  B  and  C  methods  are  not  required  to  meet  the  sufficiency 
criteria  for  coverage  of  the  practice,  the  organization,  and  the  relevant 
life-cycle  phases.  However,  the  appraisal  goals  typically  suggest  one 
emphasis  or  another.  For  example,  in  some  situations  it  is  more 
important  to  be  confident  that  all  weaknesses  have  been  identified, 
even  if  some  strengths  remain  undetected.  In  other  situations,  it  is 
important  to  identify  all  evidence  that  supports  positive  conclusions, 
even  if  minor  issues  remain  unexplored.  The  priority  given  to  one  or 
more  criteria,  in  pursuing  data  sufficient  to  support  the  appraisal 
outputs,  must  be  made  known  with  the  appraisal  outputs. 

Another  important  consideration  relating  to  data  sufficiency  revolves 
around  the  use  of  characterizations  as  an  appraisal  output.  If  instance- 
level  characterizations  are  to  be  reported  in  a  SCAMPI  B,  it  may  be 
necessary  to  examine  more  than  the  minimum  of  one  item  of  objective 
evidence  per  practice.  In  situations  where  the  instance-level 
characterizations  are  not  treated  as  intermediate  work  products 
available  only  to  the  team,  it  is  strongly  recommended  that  a  second 
item  of  objective  evidence  (e.g.,  an  indirect  artifact)  be  examined  for 
each  practice. 
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2.5  Generate  Appraisal  Results  (continued) 


Guidance: 
Improvement  Task 
Completion  Status 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Guidance  for  the 
External  Auditor 


Appraisals  can  be  used  to  monitor  ongoing  model-based  process 
improvement  efforts.  This  strategy  entails  using  the  work  breakdown 
structure  that  defines  process  improvement  tasks  as  the  reference 
model.  Because  the  improvement  program  is  model-based,  data 
supporting  conclusions  of  progress  are  also  relevant  to  the  model 
underlying  the  improvement  program.  Task  completion,  or  summaries 
expressed  as  percentages,  in  each  area  of  work  can  be  presented  in  a 
profile  with  the  appraisal  results. 


Use  of  profiles  to  communicate  a  summary  view  is  beneficial  when 
comparing  appraisal  results  from  successive  appraisals.  The  internal 
change  agent  typically  has  knowledge  of  continuity  that  helps  to 
explain  the  progression  reflected  in  the  set  of  profiles.  Also,  the 
significant  findings  or  patterns  of  characterizations  can  be  related  to 
areas  of  focus  for  the  organization’s  process  improvement  effort.  For 
example,  a  special  team  or  task  force  may  be  identified  as  the  impetus 
for  the  change  reflected  in  several  categories  of  appraisal  outputs. 


Tools  for  summarizing  and  presenting  results  may  be  one  of  the  ways 
that  a  professional  consultant  may  differentiate  himself  or  herself  from 
the  competition.  Professional  consultants  typically  bring  a  diversity  of 
previous  experiences  to  their  ability  to  explain  appraisal  outcomes. 
They  can  make  insightful  connections  between  weaknesses  and  the 
resulting  business  consequences,  as  well  as  to  the  options  available  to 
remedy  weaknesses. 


Often,  the  external  auditor  is  given  a  reporting  format  to  be  used. 
These  data-reporting  requirements  may  be  unseen  by  the  appraisal 
participants  in  some  settings.  In  other  settings,  the  external  auditor  has 
some  freedom  to  tailor  a  reporting  format  to  provide  added  benefit  to 
the  appraised  organization. 
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Phase  3 

Report  Results 

Overview 

The  minimum  requirements  for  the  process  of  reporting  results  and  the 
required  artifacts  to  be  produced  are  specified  in  this  section. 

3.1  Deliver  Appraisal 
Results 

Appraisal  results  must  be  delivered  to  designated  recipients,  including 
the  sponsor  and  stakeholders  identified  on  the  appraisal  plan. 

3.2  Package  and 
Archive  Appraisal 
Assets 

The  required  contents  of  the  appraisal  record  must  be  assembled  and 
archived.  The  sponsor  has  complete  discretion  over  access  and 
distribution  of  the  appraisal  record. 

The  CMMI  Steward  maintains  the  Appraisal  Program,  and  a  database 
of  records  from  process  appraisals.  The  required  artifacts  submitted  to 
the  steward  are  used  to  monitor  trends  and  support  quality  assurance 
activities. 
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3.1  Deliver  Appraisal  Results 


Overview 


Required  Practices 


Parameters  and  Limits: 
General 


Guidance:  Executive 
Session 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Appraisal  results  are  presented  or  delivered  to  groups  and  individuals 
designated  by  the  sponsor  and  agreed  upon  during  the  process  of 
planning  the  appraisal. 


The  appraisal  team  leader  (or  appraisal  team  members,  if  a  team  is 
used)  shall 

•  Present,  deliver  or  otherwise  transmit  designated  appraisal  results 
to  the  stakeholders  identified  in  the  appraisal  plan 

•  Reinforce,  through  written  and/or  oral  communication,  the 
provisions  in  place  for  confidentiality  of  the  appraisal  results 


Every  SCAMPI  B  and  C  appraisal  must  have  documented  results  that 
represent  a  lasting  record  of  the  outputs.  An  oral  presentation  alone  is 
not  sufficient. 

At  a  minimum,  prose  statements  about  the  approach  or  deployment  of 
practices — related  to  model  components  in  the  scope  of  the  appraisal — 
must  be  included  in  the  appraisal  results. 


It  is  common  practice  to  provide  a  forum  for  the  appraisal  sponsor  to 
discuss  the  appraisal  results  in  a  closed  session.  The  purpose  of  the 
executive  session  is  to  allow  the  sponsor  (and  other  attendees 
designated  by  the  sponsor)  to  ask  questions  and  seek  clarification  they 
might  prefer  not  to  cover  in  a  more  public  setting.  It  is  very  important 
to  emphasize  that  the  confidentiality  provisions  established  during 
planning  are  still  in  effect. 


Interaction  with  the  appraisal  participants  in  a  final  findings 
presentation  can  aid  in  developing  an  understanding  of  the  findings. 
The  findings  presentation  can  also  focus  on  next  steps,  including  the 
generation  of  corrective  action  plans  and  any  tie-in  to  future  appraisals. 
The  internal  change  agent  can  use  this  forum  as  another  element  of  the 
communication  strategy  for  an  ongoing  process  improvement  program. 


The  findings  of  a  professional  consultant  can  often  be  viewed  as  an 
external  validation  of  known  issues  in  the  organization.  Skillful 
delivery  of  final  findings  can  create  momentum  for  change,  based  on 
an  objective  view  of  the  current  state.  In  this  context,  this  external 
person  may  be  in  a  good  position  to  deliver  bad  news  as  well.  The 
perception  that  the  consultant  is  a  neutral  observer  is  very  important. 
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3.1  Deliver  Appraisal  Results  (continued) 


Guidance  for  the 
External  Auditor 


In  some  acquisition  settings,  regulations  may  preclude  delivering  final 
findings  until  after  contract  award,  if  ever.  Typically  this  would  be 
done  in  a  written  report  versus  a  presentation.  The  sponsor  of  the 
appraisal  is  typically  an  external  authority.  The  sponsor  will  designate 
who  receives  the  appraisal  results. 
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3.2  Package  and  Archive  Appraisal  Assets 


Overview 


Required  Practices 

ARC.4.7.3 

ARC.4.7.4 

Parameters  and  Limits: 
General 

ARC.4.1.2.e 


Parameters  and  Limits: 
Required  Contents  of 
the  Appraisal  Record 

ARC4.2. 16.(3) 
ARC4.2.16.a-b 
ARC4.2.16.d-e 


Parameters  and  Limits: 
Disposition  of  the 
Appraisal  Record 

ARC4.2.16(2) 


The  appraisal  sponsor  owns  the  appraisal  record,  and  controls  the 
distribution  and  dissemination  of  information  contained  in  the  record. 
This  artifact  is  the  lasting  record  for  appraisal  data. 


The  appraisal  team  leader  shall 

•  Assemble  the  appraisal  record  for  the  appraisal  sponsor 

•  Assemble  the  data  package  to  submit  to  the  CMMI  Steward 


Unless  precluded  by  the  sponsor,  the  following  items  must  be 
submitted  to  the  CMMI  Steward: 

•  appraisal  plan,  including  the  appraisal  input 

•  appraisal  findings,  including  strength  and/or  weakness  statements 

•  appraisal  disclosure  statement 

The  appraisal  findings  and  appraisal  disclosure  statement  are 
frequently  contained  in  a  final  findings  presentation.  In  such  cases, 
submitting  the  briefing  slides  (if  all  relevant  data  are  contained  in 
them),  along  with  the  appraisal  plan,  will  be  considered  sufficient  to 
meet  this  requirement. 


The  appraisal  record  must  contain  the  following: 

•  dates  of  the  appraisal 

•  appraisal  input 

•  identification  of  the  appraisal  method  used  (  including  tailoring) 

•  findings,  including  strength  and/or  weakness  statements 

•  practice  characterizations  (if  generated) 

•  other  characterizations  of  data  or  attributes  of  practices  or  projects, 
generated  during  the  appraisal  (if  any) 

•  appraisal  disclosure  statement 


Distribution  of  the  appraisal  record  to  parties  other  than  the  appraisal 
sponsor  must  be  specified  in  the  appraisal  plan  in  advance,  or  managed 
entirely  under  the  authority  of  the  appraisal  sponsor,  following 
conclusion  of  the  appraisal. 

After  the  appraisal  record  is  transmitted  to  the  appraisal  sponsor  (in 
paper  and/or  electronic  format),  participants  in  the  appraisal  process  no 
longer  have  the  ability  to  change  its  contents;  the  appraisal  process  is 
considered  complete. 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Guidance  for  the 
Internal  Change  Agent 


Guidance  for  the 

Professional 

Consultant 


Guidance  for  the 
External  Auditor 


Maintaining  records  of  process  appraisals  is  typically  an  ongoing  part 
of  a  process  improvement  program.  Often  a  library  of  documentation 
related  to  the  process  appraisal  will  be  maintained.  Review  of  past 
accomplishments  and  the  pace  of  improvement  can  be  facilitated  by 
such  a  library. 


Professional  consultants  sometimes  use  the  quality  of  the  final  report 
or  linkage  to  follow-on  activities  to  differentiate  themselves  from  the 
competition.  Using  the  appraisal  outputs  to  make  these  links  is 
addressed  during  the  planning  activities,  in  order  to  eliminate 
ambiguity  about  the  use  or  disposition  of  key  appraisal  outputs. 


The  external  auditor  will  typically  transmit  the  appraisal  results  to  a 
third  party  outside  the  appraised  organization.  This  may  require 
conformance  to  standards  in  acquisition  policies  or  other  directives. 
Avoiding  the  inadvertent  release  of  “competition  sensitive”  materials  is 
paramount. 
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Appendix  A  ARC  Traceability:  SCAMPI  A,  B,  and  C 


The  table  below  provides  a  tracing  between  the  Appraisal  Requirements  for  CMMI  (ARC)  (Version  1.1)  and  the  documentation  for 
SCAMPI  A  (in  the  column  labeled  “SCAMPI  A  MDD”),  as  well  as  the  two  methods  documented  in  this  handbook  (in  the  columns 
labeled  “SCAMPI  B”  and  “SCAMPI  C”). 

The  following  conventions  are  used  to  reduce  the  length  of  this  appendix. 

■  “P&L”  is  used  to  stand  for  “Parameters  and  Limits” 

■  “RP”  is  used  to  stand  for  “Required  Practices” 


ARC  ID 

ARC  Requirement 

SCAMPI  A  MDD 

SCAMPI  B 

SCAMPI  C 

4 

Requirements  for  CMMI  Appraisal  Methods 

4.1 

Responsibilities 

4.1.1 

The  method  shall  define  the  responsibilities  of  the 
appraisal  sponsor,  which  at  a  minimum  shall  include 
the  following  activities: 

1.2  Develop  Appraisal  Plan:  P&L: 
Sponsor  Responsibilities 

1.2  Develop  Appraisal  Plan:  P&L: 
Sponsor  Responsibilities 

4.1. l.a 

(ABC)  Verify  that  the  appraisal  team  leader  has  the 
appropriate  experience,  knowledge,  and  skills  to  take 
responsibility  for  and  lead  the  appraisal. 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  Appraisal  Input 
item  h 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  Appraisal  Input 
item  h 

4.1. l.b 

(ABC)  Ensure  that  the  appropriate  organizational  units 
or  subunits  (e.g.,  projects,  functional  units)  participate 
in  the  appraisal. 

1 . 1  Analyze  Requirements 
1.1.3  Determine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

4.1.1.C 

(ABC)  Support  appraisal  method  provisions  for 
ensuring  non-attribution  to  appraisal  participants. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L: 
Confidentiality  and  Non-Attribution  of 
Data  Sources 

1.1  Analyze  Requirements:  P&L: 
Confidentiality  and  Non-Attribution  of 
Data  Sources 
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ARC  ID 

ARC  Requirement 

SCAMPI  A  MDD 

SCAMPI  B 

SCAMPI  C 

4.1.  l.d 

(ABC)  Ensure  that  resources  are  made  available  to 
conduct  the  appraisal. 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.1  Analyze  Requirements:  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements:  P&L: 
Appraisal  Constraints 

4.1. l.e 

(ABC)  Review  and  approve  the  appraisal  input  prior  to 
the  beginning  of  data  collection  by  the  appraisal  team. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  RP 

1.1  Analyze  Requirements:  RP 

4.1.2 

The  method  shall  define  the  responsibilities  of  the 
appraisal  team  leader,  which  at  a  minimum  shall 
include  the  following  activities: 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

Chapter  2  Introduction:  Appraisal 
Processes 

Chapter  2  Introduction:  Appraisal 
Processes 

4. 1.2. a 

(ABC)  Ensure  that  the  appraisal  is  conducted  in 
accordance  with  the  method’s  documented  process. 

1 .2  Develop  Appraisal  Plan 
1.2.1  Tailor  Method 

Chapter  2  Introduction:  Appraisal 
Processes 

Chapter  2  Introduction:  Appraisal 
Processes 

4.1.2.b 

(ABC)  Confirm  the  sponsor’s  commitment  to  proceed 
with  the  appraisal. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1 .2  Develop  Appraisal  Plan:  RP 

1.2  Develop  Appraisal  Plan:  RP 

4.1.2.C 

(ABC)  Ensure  that  appraisal  participants  are  briefed  on 
the  purpose,  scope,  and  approach  of  the  appraisal. 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.1.2.d 

(ABC)  Ensure  that  all  appraisal  team  members  have 
the  appropriate  experience,  knowledge,  and  skills  in  the 
appraisal  reference  model  and  appraisal  method;  the 
necessary  competence  to  use  instruments  or  tools 
chosen  to  support  the  appraisal;  and  access  to 
documented  guidance  on  how  to  perform  the  defined 
appraisal  activities. 

1 .3  Select  and  Prepare  Team 

1.3.3  Prepare  Team 

1 .3  Select  and  Prepare  Team:  RP 

1 .3  Select  and  Prepare  Team:  RP 

4.1.2.e 

(ABC)Verify  and  document  that  the  appraisal  method 
requirements  have  been  met. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L 

4.2 

Appraisal  Method  Documentation 

4.2.1 

The  method  shall  be  documented  and,  at  a  minimum, 
include 

MDD,  VI. 1 

All 

This  Handbook 

This  Handbook 
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ARC  ID 

ARC  Requirement 

SCAMPI  A  MDD 

SCAMPI  B 

SCAMPI  C 

4.2. l.a 

(ABC)  identification  of  the  CMMI  models  (version, 
discipline,  and  representation  [staged  or  continuous]) 
with  which  the  method  can  be  used 

Method  Overview 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

4.2. l.b 

(ABC)  identification  of  the  ARC  version  upon  which 
the  appraisal  method  is  based 

Method  Context 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

4.2. l.c 

(ABC)  identification  of  which  CMMI  appraisal 
requirements  are  satisfied  by  the  method,  along  with 
the  CMMI  appraisal  class  membership  (if  applicable) 

Method  Context 

This  Table 

This  Table 

4.2. l.d 

(ABC)  activity  descriptions,  artifacts,  and  guidance 
that  implement  each  of  the  appraisal  requirements 

(All  phases,  processes, 
activities) 

This  Handbook 

This  Handbook 

4.2. l.e 

(A)  declaration  as  to  whether  or  not  the  method 
supports  15504-conformant  appraisals 

TBD 

4.2.2 

The  method  documentation  shall  provide  guidance  for 

4.2. 2. a 

(ABC)  identifying  an  appraisal's  purpose,  objectives, 
and  constraints 

1 . 1  Analyze  Requirements 

1.1.1  Determine  Appraisal 
Objectives 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  Appraisal  Input 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  Appraisal  Input 

4.2.2.b 

(ABC)  determining  the  suitability  of  the  appraisal 
method  relative  to  the  appraisal’s  purpose,  objectives, 
and  constraints 

1 . 1  Analyze  Requirements 

1.1.1  Determine  Appraisal 
Objectives 

1.1  Analyze  Requirements:  P&L 

1.1  Analyze  Requirements:  P&L 

4.2.3 

The  method  documentation  shall  provide  guidance  for 
identifying  the  scope  of  the  CMMI  model(s)  to  be  used 
for  the  appraisal: 

1 . 1  Analyze  Requirements 
1.1.3  Dctennine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

4.2. 3. a 

(ABC)  process  areas  to  be  investigated  (continuous  and 
staged  representations) 

1 . 1  Analyze  Requirements 
1.1.3  Dctennine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

4.2. 3.b 

(ABC)  capability  levels  to  be  investigated  for  each 
process  area  (continuous  representation) 

1 . 1  Analyze  Requirements 
1.1.3  Dctennine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

1.1  Analyze  Requirements:  P&L: 

Model  Scope 

4.2.4 

The  method  documentation  shall  provide  guidance  for 
identifying  the  organizational  unit  to  be  appraised: 

1 . 1  Analyze  Requirements 
1.1.3  Dctennine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

4.2. 4. a 

(ABC)  the  sponsor  of  the  appraisal  and  the  sponsor’s 
relationship  to  the  organizational  unit  being  appraised 

1 . 1  Analyze  Requirements 

1.1.1  Dctennine  Appraisal 
Goals 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 
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4.2.4.b 

(ABC)  projects  within  the  organizational  unit  that  will 
participate 

1 . 1  Analyze  Requirements 
1.1.3  Determine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

4.2.4.C 

(ABC)  functional  elements  of  the  organizational  unit 
that  will  participate 

1 . 1  Analyze  Requirements 
1.1.3  Determine  Appraisal 
Scope 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

4.2.4.d 

(ABC)  names  and  affiliations  (organizational  units)  of 
participants  in  the  appraisal  activities 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

1.1  Analyze  Requirements:  P&L: 
Organizational  Unit 

4.2.5 

The  method  documentation  shall  provide  guidance  for 
selecting  appraisal  team  members  and  criteria  for 
qualification  including 

1 .3  Select  and  Prepare  Team 

1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

4.2. 5. a 

(ABC)  technical  experience  (discipline-specific) 

1 .3  Select  and  Prepare  Team 
1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

4.2. 5.b 

(ABC)  management  experience 

1 .3  Select  and  Prepare  Team 
1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

4.2. 5.c 

(ABC)  experience,  knowledge,  and  skills  in  the 
appraisal  reference  model  and  appraisal  method 

1 .3  Select  and  Prepare  Team 
1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

4.2.6 

The  method  documentation  shall  provide  guidance  for 
an  appraisal  team  leader’s  qualification  criteria, 
including 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

4.2. 6.a 

(ABC)  training  and  experience  using  the  appraisal 
reference  model 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

4.2. 6.b 

(ABC)  training  and  experience  using  the  appraisal 
method 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

4.2. 6. c 

(ABC)  experience  in  delivering  training,  managing 
teams,  facilitating  group  discussions,  and  making 
presentations 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Appraisal  (Team)  Leader  Qualifications 

4.2.7 

(ABC)  The  method  documentation  shall  provide 
guidance  for  detennining  the  appropriate  size  of  the 
appraisal  team. 

1 .3  Select  and  Prepare  Team 
1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Qualifications 
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4.2.8 

(ABC)  The  method  documentation  shall  provide 
guidance  on  the  roles  and  responsibilities  of  appraisal 
team  members. 

1 .3  Select  and  Prepare  Team 
1.3.2  Select  Team  Members 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Roles  and 
Responsibilities 

Guidance:  Use  of  Mini-teams 

1.3  Select  and  Prepare  Team:  P&L: 
Team  Member  Roles  and 
Responsibilities 

Guidance:  Use  of  Mini-teams 

4.2.9 

(ABC)  The  method  documentation  shall  provide 
guidance  addressing  the  responsibilities  of  the 
appraisal  sponsor. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1 .2  Develop  Appraisal  Plan:  P&L 
Sponsor  Responsibilities 

1.2  Develop  Appraisal  Plan:  P&L 
Sponsor  Responsibilities 

4.2.10 

(ABC)  The  method  documentation  shall  provide 
guidance  addressing  the  responsibilities  of  the 
appraisal  team  leader. 

1 .3  Select  and  Prepare  Team 
1.3.1  Identify  Team  Leader 

Chapter  2:  Appraisal  Processes 

Chapter  2:  Appraisal  Processes 

4.2.11 

(ABC)  The  method  documentation  shall  provide 
guidance  for  estimating  the  resources  required  to 
conduct  the  appraisal  (including  the  amount  of  time 
required  to  conduct  an  appraisal). 

1 .2  Develop  Appraisal  Plan 
1.3.1  Identify  Team  Leader 

Chapter  1,  Executive  Summary 

Rough  Order  of  Magnitude  Estimating 

Chapter  1 ,  Executive  Summary 

Rough  Order  of  Magnitude  Estimating 

4.2.12 

(ABC)  The  method  documentation  shall  provide 
guidance  for  appraisal  logistics. 

1 .2  Develop  Appraisal  Plan 

1 .2.4  Plan  and  Manage 
Logistics 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal  Plan 

4.2.13 

(ABC)  The  method  documentation  shall  provide 
guidance  for  collecting  relevant  data  on  the 
organizational  unit  and  associating  the  data  to  the 
specific  and  generic  practices  of  the  appraisal  reference 
model. 

2.1  Examine  Objective 
Evidence 

2.1  Examine  Objective  Evidence:  RP 

2.1  Examine  Objective  Evidence:  RP 

4.2.14 

(ABC)  The  method  documentation  shall  provide 
guidance  for  creating  findings,  including  both  strengths 
and  weaknesses  relative  to  the  appraisal  reference 
model. 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.5  Generate  Appraisal  Results:  P&L 
P&L  Findings 

2.5  Generate  Appraisal  Results:  P&L 
P&L  Findings 
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4.2.15 

(ABC)  The  method  documentation  shall  provide 
guidance  for  protecting  the  confidentiality  of  appraisal 
data  and  ensuring  non-attribution  of  data  contributed 
by  appraisal  participants. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.4  Archive  and/or  Dispose 
of  Key  Artifacts 

1.1  Analyze  Requirements:  P&L: 
Confidentiality  and  Non-attribution  of 
Data  Sources 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence  :  P&L: 
Confidentiality  and  Non-attribution  of 
Data  Sources 

2.2  Document  Objective  Evidence: 

P&L:  Non-attribution 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Non-Attribution 

1.1  Analyze  Requirements:  P&L: 
Confidentiality  and  Non-attribution  of 
Data  Sources 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Confidentiality  and  Non-attribution  of 
Data  Sources 

2.2  Document  Objective  Evidence: 

P&L:  Non-attribution 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Non-Attribution 

4.2.16 

The  method  documentation  shall  provide  guidance  for 
(1)  recording  traceability  between  the  data  collected 
during  the  appraisal  and  the  findings  and/or  ratings, 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

2.2  Document  Objective  Evidence: 

P&L:  Traceability 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Model  Relevance 

2.2  Document  Objective  Evidence: 

P&L:  Traceability 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Model  Relevance 

(2)  the  retention  and  safekeeping  of  appraisal  records, 
and 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Disposition  of  appraisal 
record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Disposition  of  appraisal 
record 

(3)  compiling  and  maintaining  an  appraisal  record  that 
supports  the  appraisal  team’s  findings  and/or  ratings 
and  that  contains  the  following  minimum  content: 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

4.2.16.a 

(ABC)  dates  of  appraisal 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

3.2  Package  and  Archive  Appraisal 
Assets  :  P&L:  Required  Contents  of  the 
appraisal  record 

4.2.16.b 

(ABC)  appraisal  input 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

4.2.16.C 

(A)  objective  evidence,  or  identification  thereof, 
sufficient  to  substantiate  goal  rating  judgments 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 
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4.2.16.d 

(ABC)  identification  of  appraisal  method  (and  version) 
used,  along  with  any  tailoring  options 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

4.2.16.e 

(ABC)  findings 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

3.2  Package  and  Archive  Appraisal 
Assets:  P&L:  Required  Contents  of  the 
appraisal  record 

4.2. 1 6. f 

(A)  any  ratings  rendered  during  the  appraisal  (goals, 
process  areas,  and  maturity  or  capability  levels) 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

4.2.16.g 

(A)  the  set  of  15504  process  profiles  resulting  from  the 
appraisal,  if  requested  by  the  appraisal  sponsor 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

4.3 

Planning  and  Preparing  for  the  Appraisal 

Chapter  2:  1 .  Plan  and  Prepare  for 
Appraisal 

Chapter  2:  1.  Plan  and  Prepare  for 
Appraisal 

4.3.1 

The  method  shall  provide  for  the  preparation  of 
appraisal  participants  by  addressing,  at  a  minimum, 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1.4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.3. l.a 

(ABC)  the  purpose  of  the  appraisal 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.3. l.b 

(ABC)  the  scope  of  the  appraisal 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.3. l.c 

(ABC)  the  appraisal  approach 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 
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4.3. l.d 

(ABC)  the  roles  and  responsibilities  of  participants  in 
the  appraisal 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.3. l.e 

(ABC)  the  schedule  of  appraisal  activities 

1 .4  Obtain  and  Analyze 
Preliminary  Objective 
Evidence 

1 .4. 1  Prepare  Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

1 .4  Prepare  Participants  and  Obtain 
Initial  Objective  Evidence:  P&L: 
Communication  with  Appraisal 
Participants 

4.3.2 

(ABC)  The  method  shall  provide  for  the  development 
of  the  appraisal  input  prior  to  the  beginning  of  data 
collection  by  the  appraisal  team. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L 

1.1  Analyze  Requirements:  P&L 

4.3.3 

At  a  minimum,  the  appraisal  input  shall  specify 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3. 3. a 

(ABC)  the  identity  of  the  sponsor  of  the  appraisal  and 
the  sponsor’s  relationship  to  the  organizational  unit 
being  appraised 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements:  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.b 

(ABC)  the  appraisal  purpose,  including  alignment  with 
business  objectives 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.C 

(ABC)  the  appraisal  reference  model  scope,  including 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.C.1 

the  process  areas  to  be  investigated  within  the 
organizational  unit 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

4.3.3.C.2 

the  highest  maturity  level  and/or  capability  level  to  be 
investigated  for  each  process  area  within  the  appraisal 
scope 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

1.1  Analyze  Requirements  P&L: 

Model  Scope 

4.3.3.d 

(ABC)  the  organizational  unit  that  is  the  subject  of  the 
appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3. 3. e 

(ABC)  the  process  context,  which,  at  a  minimum,  shall 
include 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 
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4.3.3.e.l 

the  size  of  the  organizational  unit 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Process  Context 

1.1  Analyze  Requirements  P&L: 

Process  Context 

4.3.3.e.2 

the  demographics  of  the  organizational  unit 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Process  Context 

1.1  Analyze  Requirements  P&L: 

Process  Context 

4.3.3. e.3 

the  application  domain  of  the  products  or  services  of 
the  organizational  unit 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Process  Context 

1.1  Analyze  Requirements  P&L: 

Process  Context 

4.3.3.e.4 

the  size,  criticality,  and  complexity  of  the  products  or 
services 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Process  Context 

1.1  Analyze  Requirements  P&L: 

Process  Context 

4.3.3.e.5 

the  quality  characteristics  of  the  products  or  services 
(e.g.,  defect  density,  reliability) 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 

Process  Context 

1.1  Analyze  Requirements  P&L: 

Process  Context 

4.3.3.f 

(ABC)  the  appraisal  constraints,  which,  at  a  minimum, 
shall  include 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.f.l 

availability  of  key  resources  (e.g.,  staffing,  funding, 
tools,  facilities) 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.f.2 

schedule  constraints 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.f.3 

the  maximum  amount  of  time  to  be  used  for  the 
appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.f.4 

specific  process  areas  or  organizational  entities  to  be 
excluded  from  the  appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.f.5 

the  minimum,  maximum,  or  specific  sample  size  or 
coverage  that  is  desired  for  the  appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 
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4.3. 3. f.  6 

the  ownership  of  the  appraisal  outputs  and  any 
restrictions  on  their  use 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3. 3. f.  7 

controls  on  information  resulting  from  a  confidentiality 
agreement 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.f.8 

non-attribution  of  appraisal  data  to  associated  sources 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

1.1  Analyze  Requirements  P&L: 
Appraisal  Constraints 

4.3.3.g 

(ABC)  the  identity  of  the  CMMI  models  used, 
including  the  version,  discipline,  and  representation 
(staged  or  continuous) 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.h 

(ABC)  the  criteria  for  experience,  knowledge,  and 
skills  of  the  appraisal  team  leader  who  is  responsible 
for  the  appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.i 

(ABC)  the  identity  and  affiliation  of  the  appraisal  team 
members,  including  the  appraisal  team  leader,  with 
their  specific  responsibilities  for  the  appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.j 

(ABC)  the  identity  (name  and  organizational 
affiliation)  of  appraisal  participants  and  support  staff, 
with  specific  responsibilities  for  the  appraisal 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.k 

(ABC)  any  additional  information  to  be  collected 
during  the  appraisal  to  support  achievement  of  the 
appraisal  objectives 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.1 

(ABC)  a  description  of  the  planned  appraisal  outputs, 
including  ratings  to  be  generated  (process  areas, 
maturity  level) 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3. 3. m 

(ABC)  anticipated  follow-on  activities  (e.g.,  reports, 
appraisal  action  plans,  re-appraisal) 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

4.3.3.n 

(ABC)  planned  tailoring  of  the  appraisal  method  and 
associated  tradeoffs,  including  the  sample  size  or 
coverage  of  the  organizational  unit 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 

1.1  Analyze  Requirements  P&L: 
Required  Contents  of  the  Appraisal 

Input 
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4.3.4 

(ABC)  The  method  shall  require  that  the  appraisal 
input  and  any  changes  to  the  appraisal  input  shall  be 
agreed  to  by  the  sponsor  (or  the  delegated  authority) 
and  documented  in  the  appraisal  record. 

1 . 1  Analyze  Requirements 
1.1.5  Obtain  Commitment  to 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L 

P&L:  Required  Contents  of  the 
Appraisal  Input 

1.1  Analyze  Requirements:  P&L 

P&L:  Required  Contents  of  the 
Appraisal  Input 

4.3.5 

The  method  shall  require  the  development  of  an 
appraisal  plan  that,  at  a  minimum,  specifies 

4.3. 5.a 

(ABC)  the  appraisal  input 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

4.3. 5.b 

(ABC)  the  activities  to  be  performed  in  conducting  the 
appraisal 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

4.3.5.C 

(ABC)  resources  and  schedule  assigned  to  appraisal 
activities 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

4.3.5.d 

(ABC)  appraisal  logistics 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

4.3.5.e 

(ABC)  mitigation  steps  to  address  risks  associated  with 
appraisal  execution 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

1.2  Develop  Appraisal  Plan:  P&L: 
Required  Contents  of  the  Appraisal 
Record 

4.3.5.f 

(A)  the  criteria  to  verify  that  the  requirements  of 
ISO/IEC  15504  have  been  met,  if  requested  by  the 
appraisal  sponsor 

1 .2  Develop  Appraisal  Plan 
1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

4.4 

Appraisal  Data  Collection 

4.4. intro 

Appraisal  teams  base  their  findings  on  observations 
that,  in  turn,  are  based  on  objective  evidence  gathered 
from  one  or  more  sources.  The  requirements  in  this 
section  identify  the  sources  of  objective  evidence 
recognized  by  CMMI  appraisal  methods.  As  indicated 
in  Appendix  A,  all  three  sources  of  objective  evidence 
identified  below  are  required  for  Class  A  appraisal 
methods.  At  least  two  sources  are  required  for  Class  B 
methods,  one  of  which  must  be  interviews.  At  least  one 
source  is  required  for  Class  C  methods. 

2.1  Examine  Objective  Evidence  P&L: 
Objective  Evidence 

2.1  Examine  Objective  Evidence  P&L: 
Objective  Evidence 
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4.4.1 

(See  Appendix  A)The  method  shall  collect  data  by 
administering  instruments  (e.g.,  questionnaires, 
surveys). 

2.1  Examine  Objective 
Evidence 

2.1.1  Examine  Objective 
Evidence  from  Instruments 

2.1  Examine  Objective  Evidence: 
Guidance:  Instruments 

2.1  Examine  Objective  Evidence: 
Guidance:  Instruments 

4.4.2 

(See  Appendix  A)  The  method  shall  collect  data  by 
conducting  interviews  (e.g.,  with  project  leaders, 
managers,  practitioners). 

2.1  Examine  Objective 
Evidence 

2.1.4  Examine  Objective 
Evidence  from  Interviews 

2.1  Examine  Objective  Evidence:  P&L: 
Interviews 

2.1  Examine  Objective  Evidence:  P&L: 
Interviews 

4.4.3 

(See  Appendix  A)  The  method  shall  collect  data  by 
reviewing  documentation  (e.g.,  organizational  policies, 
project  procedures,  and  implementation-level  work 
products). 

2.1  Examine  Objective 
Evidence 

2.1.3  Examine  Objective 
Evidence  from  Documents 

2.1  Examine  Objective  Evidence:  P&L: 
Work  Actually  Being  Done 

2.1  Examine  Objective  Evidence:  P&L: 
Work  Actually  Being  Done 

4.5 

Data  Consolidation  and  Validation 

4.5.1 

(AB)  The  method  shall  require  appraisal  team 
consensus  in  decisions  when  determining  the  validity 
of  observations,  creating  findings,  and  establishing 
ratings. 

2.4  Generate  Appraisal 

Results 

2.4.1  Derive  Findings  and 

Rate  Goals 

2.3  Verify  Objective  Evidence:  RP 

P&L:  Verify  Model  Relevance 

2.5  Generate  Appraisal  Results:  P&L 

4.5.2 

The  method  shall  require  a  mechanism  for 
consolidating  the  data  collected  during  an  appraisal 
into  accurate  observations  according  to  the  following 
criteria: 

4. 5. 2. a 

(ABC)  The  observation  was  derived  from  objective 
evidence  seen  or  heard  during  data  collection  sessions. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation  Gaps 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Accuracy 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Accuracy 

4.5.2.b 

(ABC)  The  observation  is  clearly  worded,  phrased 
without  attribution,  and  expressed  in  terminology  used 
at  the  organizational  unit. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation  Gaps 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Non-attribution 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Non-attribution 

4.5.2.C 

(ABC)  The  observation  is  relevant  to  the  appraisal 
reference  model  and  can  be  associated  with  a  specific 
model  component. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation  Gaps 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Model  Relevance 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Model  Relevance 

94 


CMU/SEI-2005-HB-005 


ARC  ID 

ARC  Requirement 

SCAMPI  A  MDD 

SCAMPI  B 

SCAMPI  C 

4.5.3 

The  method  shall  require  a  mechanism  for  validating 
each  accurate  observation  according  to  the  following 
criteria: 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verify  Objective  Evidence  (all) 

4.5.3.a 

(AB)  The  observation  is  corroborated. 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Corroboration 

4.5.3.b 

( AB)  The  observation  is  consistent  with  other  validated 
observations.  (Validated  observations  camiot  be  both 
tine  and  mutually  inconsistent;  in  aggregate,  they 
constitute  a  set  of  truths  about  the  organizational  unit 
that  must  be  consistent.) 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Consistency 

4.5.4 

The  method  shall  require  the  following  minimum  set  of 
criteria  to  be  satisfied  in  order  for  an  observation  to  be 
considered  “corroborated”: 

4.5.4.a 

(AB)  The  observation  is  based  on  data  from  at  least 
two  different  sources  (e.g.,  the  data  should  originate 
from  at  least  two  different  individuals). 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Corroboration 

4.5.4.b 

(AB)  The  observation  is  based  on  data  from  at  least 
two  different  data-gathering  sessions. 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verify  Objective  Evidence:  P&L: 
Verify  Corroboration 

4.5.4.C 

( AB)  At  least  one  of  the  two  data  points  must  reflect 
work  actually  being  done  (e.g.,  process  area 
implementation). 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

2.3  Verily  Objective  Evidence:  P&L: 
Work  Actually  Being  Done 

4.5.5 

The  method  shall  require  a  mechanism  for  determining 
that  sufficient  data  has  been  collected  to  cover  the 
scope  of  the  appraisal,  according  to  the  following 
minimum  set  of  rules: 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 
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4.5. 5. a 

(A)  A  specific  or  generic  practice  has  sufficient  data 
coverage  if  validated  observations  exist  for  the  practice 
and 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5.5.a.l 

are  adequate  to  understand  the  extent  of 
implementation  of  the  practice 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5.5.a.2 

are  representative  of  the  organizational  unit 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5.5.a,3 

are  representative  of  the  life-cycle  phases  in  use  within 
the  organizational  unit 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5. 5.b 

(A)  In  a  staged  representation,  a  process  area  has 
sufficient  data  coverage  if  all  of  its  specific  and  generic 
practices  have  sufficient  data  coverage. 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5.5.C 

(A)  In  a  continuous  representation,  a  process  area  has 
sufficient  data  coverage  if  all  of  its  specific  practices 
and  the  generic  practices  within  the  appraisal  scope 
have  sufficient  data  coverage  up  through  the  capability 
level  being  investigated  for  the  process  area  (e.g.,  the 
target  capability  level). 

2.2  Verify  and  Validate 
Objective  Evidence 

2.2.1  Verify  Objective 
Evidence 

4.5.6 

(A)  The  method  shall  require  a  mechanism  for 
consolidating  observations  into  draft  findings  of 
strengths  and  weaknesses  relative  to  the  appraisal 
reference  model. 

2.2  Verify  and  Validate 
Objective  Evidence 

3.7.3  Validate  Practice 
Implementation  Gaps 

4.5.7 

(A)  The  method  shall  require  that  the  appraisal 
participants  be  presented  with  the  draft  findings  in 
order  to  solicit  their  responses  for  verification  of  the 
findings’  accuracy  and  clarity. 

2.2  Verify  and  Validate 
Objective  Evidence 

3.7.3  Validate  Practice 
Implementation  Gaps 

2.4  Validate  Preliminary  Appraisal 
Outputs 

2.4  Validate  Preliminary  Appraisal 
Outputs 

4.6 

Rating 
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4.6.1 

The  method  shall  define  a  rating  process  that  specifies, 
at  a  minimum,  the  following: 

2.4  Generate  Appraisal 

Results 

4.6. l.a 

(A)  An  appraisal  team  can  rate  a  specific  or  generic 
goal  when  valid  observations  for  each  practice  related 
to  the  goal  meet  the  method’s  defined  data  coverage 
criteria. 

2.4  Generate  Appraisal 

Results 

2.4.1  Derive  Findings  and 

Rate  Goals 

4.6. 1 .b 

(A)  An  appraisal  team  can  rate  a  process  area  when  it 
has  rated  each  of  the  process  area’s  specific  goals  and 
generic  goals  within  the  appraisal  scope. 

2.4  Generate  Appraisal 

Results 

2.4.2a  Determine  Process 

Area  Capability  Level 

3.9.2b  Determine  Satisfaction 
of  Process  Areas 

4.6.1.C 

(A)An  appraisal  team  can  detennine  a  maturity  level 
rating  once  it  has  rated  all  of  the  process  areas  within 
that  level  and  each  level  below. 

2.4  Generate  Appraisal 

Results 

2.4.3b  Determine  Maturity 
Level 

4.6. 1 .d 

(A)  An  appraisal  team  can  detennine  the  capability 
level  of  a  process  area  when  it  has  rated  each  of  the 
generic  goals  at  or  below  the  target  capability  level. 

2.4  Generate  Appraisal 

Results 

2.4.3a  Derive  Process  Area 
Capability  Profile 

4.6.2 

(A)The  method  shall  require  that  maturity  level  ratings 
and/or  capability  level  ratings  be  based  on  the 
definitions  of  capability  levels  and  maturity  levels  in 
the  CMMI  models. 

2.4  Generate  Appraisal 

Results 

2.4.3a  Derive  Process  Area 
Capability  Profile 

2.4.3b  Determine  Maturity 
Level 

4.6.3 

The  method  shall  rate  each  specific  and  generic  goal 
(provided  the  prerequisites  of  rating  have  been 
completed)  within  the  appraisal  scope  in  accordance 
with  the  following  rules: 

2.4  Generate  Appraisal 

Results 

2.4.1  Derive  Findings  and 

Rate  Goals 

4.6.3. a 

(A)  Rate  the  goal  “satisfied”  when  the  associated 
generic  or  specific  practices  (or  acceptable  alternative 
practices)  are  judged  to  be  implemented  and  the 
aggregate  of  weaknesses  does  not  have  a  significant 
negative  impact  on  goal  achievement. 

2.4  Generate  Appraisal 

Results 

2.4.1  Derive  Findings  and 

Rate  Goals 
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4.6.3.b 

(A)  Rate  the  goal  “unsatisfied”  otherwise. 

2.4  Generate  Appraisal 

Results 

2.4.1  Derive  Findings  and 

Rate  Goals 

4.6.4 

The  method  shall  rate  each  process  area  within  the 
appraisal  scope,  if  requested  by  the  appraisal  sponsor, 
in  accordance  with  the  following  rules: 

4.6.4.a 

(A)  For  a  staged  representation,  the  process  area  is 
“satisfied”  if  and  only  if  all  of  its  specific  and  generic 
goals  are  rated  “satisfied.” 

2.4  Generate  Appraisal 

Results 

2.4.2b  Determine  Satisfaction 
of  Process  Areas 

4.6.4.b 

(A)  For  a  continuous  representation,  the  process  area  is 
given  a  capability  level  rating  based  upon  the  highest 
level  and  all  levels  below  for  which  its  specific  goals 
and  the  generic  goals  within  the  appraisal  scope  have 
been  satisfied. 

2.4  Generate  Appraisal 

Results 

2.4.2a  Determine  Process 

Area  Capability  Level 

4.6.4.C 

(A)  When  a  process  area  is  determined  to  be  outside  of 
the  organizational  unit’s  scope  of  work,  the  process 
area  is  designated  as  “not  applicable”  and  is  not  rated. 

2.4  Generate  Appraisal 

Results 

2.4.3a  Determine  Process 

Area  Capability  Profile 

2.4.3b  Determine  Maturity 
Level 

4.6.4.d 

(A)  When  a  process  area  is  outside  of  the  appraisal 
scope,  or  if  the  associated  findings  do  not  meet  the 
method’s  defined  criteria  for  data  coverage,  the  process 
area  is  designated  as  “not  rated”  and  is  not  rated. 

2.4  Generate  Appraisal 

Results 

2.4.3a  Determine  Process 

Area  Capability  Profile 

2.4.3b  Determine  Maturity 
Level 

4.6.5 

The  method  shall  rate  the  maturity  level,  if  requested 
by  the  appraisal  sponsor,  in  accordance  with  the 
following  rules: 

2.4  Generate  Appraisal 

Results 

2.4.3b  Determine  Maturity 
Level 
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ARC  Requirement 

SCAMPI  A  MDD 

SCAMPI  B 

SCAMPI  C 

4.6.5. a 

(A)  A  maturity  level  for  a  staged  representation  is 
achieved  if  all  process  areas  within  the  level  and  within 
each  lower  level  are  either  “satisfied”  or  “not 
applicable.” 

2.4  Generate  Appraisal 

Results 

2.4.3b  Determine  Maturity 
Level 

4.6.5.b 

(A)  A  maturity  level  for  a  continuous  representation  is 
achieved  if  the  capability  level  profile  is  at  or  above  the 
target  profile  for  all  process  areas  for  that  maturity 
level  and  all  lower  maturity  levels  in  the  equivalent 
staging,  excepting  those  process  areas  that  are 
designated  as  “not  applicable.” 

2.4  Generate  Appraisal 

Results 

2.4.3b  Determine  Maturity 
Level 

4.7 

Reporting  Results 

4.7.1 

(ABC)  The  method  shall  require  documenting  and 
reporting  the  appraisal  findings  and/or  ratings  to  the 
appraisal  sponsor  and  to  the  appraised  organization. 

3.1  Deliver  Appraisal 

Results 

3.1.1  Present  Final  Findings 

3.1  Deliver  Appraisal 

Results:  RP 

3.1  Deliver  Appraisal 

Results:  RP 

4.7.2 

(A)  If  ISO/IEC  15504  conformance  is  desired,  the 
method  shall  define  a  mechanism  for  converting 
objective  evidence  used  by  the  appraisal  team  as  the 
basis  for  goal  ratings  into  associated  process  attribute 
outcomes  in  accordance  with  the  translation 
requirement  of  ISO/IEC  TR  15504-2  (clause  7.6). 

TBD 

4.7.3 

(A)  The  method  shall  require  the  submission  of 
appraisal  data  required  by  the  CMMI  Steward  for  the 
purpose  of  reporting  aggregated  appraisal  information 
to  the  constituent  community. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.3  Provide  Appropriate 
Feedback  to  CMMI  Steward 

3.2  Package  and  Archive  Appraisal 
Assets 

3.2  Package  and  Archive  Appraisal 
Assets 

4.7.4 

(ABC)  The  method  shall  require  that  the  appraisal 
record  be  provided  to  the  appraisal  sponsor  for 
retention. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

3.2  Package  and  Archive  Appraisal 
Assets:  RP 

3.2  Package  and  Archive  Appraisal 
Assets:  RP 
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Glossary 


Affirmation  —  An  oral  or  written  statement  confirming  or  supporting  implementation  of 
a  CMMI  model  specific  practice  or  generic  practice.  Affirmations  are  usually  provided 
by  the  implementers  of  the  practice  and/or  internal  or  external  customers,  but  may  also 
include  other  stakeholders  (e.g.,  managers,  suppliers),  [derived  from  scampi  mdd  method  overview] 
Interview  responses  are  examples  of  face-to-face  affirmations.  Alternative  forms  of 
affirmations  could  include  presentations  or  demonstrations  of  a  tool  or  mechanism  as  it 
relates  to  implementation  of  a  CMMI  model  practice,  [derived  from  scampi  mdd  pii  appendix  b] 

Alternative  Practice  —  A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific 
practices  contained  in  the  CMMI  model  that  achieves  an  equivalent  effect  toward 
satisfying  the  goal  associated  with  the  practices.  Alternative  practices  are  not  necessarily 
one-for-one  replacements  for  the  generic  or  specific  practices,  [arc  vi.i  andCMMi  model  glossary] 

Appraisal  —  An  examination  of  one  or  more  processes  by  a  trained  team  of 
professionals  using  an  appraisal  reference  model  as  the  basis  for  determining,  as  a 
minimum,  strengths  and  weaknesses,  [arcvi.i] 

Appraisal  Disclosure  Statement  (ADS)  —  A  summary  statement  describing  the  ratings 
generated  as  outputs  of  the  appraisal,  and  the  conditions  and  constraints  under  which  the 
appraisal  was  perfonned.  The  ADS  should  be  used  for  public  disclosures  of  maturity 
level  or  capability  level  ratings  so  they  can  be  interpreted  accurately,  [scampi  mdd] 

Appraisal  Input  —  The  collection  of  appraisal  information  required  before  data 
collection  can  commence,  [iso  98c  and  arc  vi.i] 

Appraisal  Method  Class  —  A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of 
requirements  in  the  Appraisal  Requirements  for  CMMI  (ARC).  These  classes  are  defined 
so  as  to  align  with  typical  usage  modes  of  appraisal  methods,  [derived  from  arc  vi.o,  cmmi  model 

glossary  and  ARC  v  1 . 1  ] 


Appraisal  Objectives  —  The  desired  outcome(s)  of  an  appraisal  process,  [arcvi.i] 

Appraisal  Outputs  —  All  of  the  tangible  results  from  an  appraisal  (see  “appraisal 
record”).  [IS0  98C  and  ARCvl.l] 

Appraisal  Participant  —  Members  of  the  organizational  unit  who  participate  in 
providing  information  during  the  appraisal,  [cmmi  model  glossary  and  arcvi.i] 
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Appraisal  Record  —  An  orderly,  documented  collection  of  infonnation  that  is  pertinent 
to  the  appraisal  and  adds  to  the  understanding  and  verification  of  the  appraisal  findings 
and  ratings  generated,  [derived  from  iso  98c  and  arc  vi.i] 

Appraisal  Reference  Model  —  The  CMMI  model  to  which  an  appraisal  team  correlates 
implemented  process  activities,  [cmmi  model  glossary  and  arc  vi.i] 

Appraisal  Sponsor  —  The  individual,  internal  or  external  to  the  organization  being 
appraised,  who  requires  the  appraisal  to  be  perfonned,  and  provides  financial  or  other  resources 

to  Carry  it  out.  [derived  from  ISO  98C  and  ARC  vl.l] 

Appraisal  Tailoring  —  Selection  of  options  within  the  appraisal  method  for  use  in  a 
specific  instance.  The  intent  of  tailoring  is  to  assist  an  organization  in  aligning  application 
of  the  method  with  its  business  needs  and  objectives,  [cmmi  model  glossary  and  arc  vi.i] 

Appraisal  Team  Leader  —  The  person  who  leads  the  activities  of  an  appraisal  and  has 
satisfied  the  qualification  criteria  for  experience,  knowledge,  and  skills  defined  by  the 
appraisal  method.  [ARCvi.i] 

Consensus  —  A  method  of  decision  making  that  allows  team  members  to  develop  a 
common  basis  of  understanding  and  develop  general  agreement  concerning  a  decision 
that  all  team  members  are  willing  to  support,  [arc  vi.i] 

Consolidation  —  The  activity  of  collecting  and  summarizing  the  infonnation  provided 
into  a  manageable  set  of  data  to  (a)  detennine  the  extent  to  which  the  data  are 
conoborated  and  cover  the  areas  being  investigated,  (b)  determine  the  data’s  sufficiency 
for  making  judgments,  and  (c)  revise  the  data-gathering  plan  as  necessary  to  achieve  this 
sufficiency.  [ARCvi.i] 

Corroboration  —  The  extent  to  which  enough  data  has  been  gathered  to  confinn  that  an 
observation  is  acceptable  for  use  by  an  appraisal  team,  [arc  vi.i]  In  SCAMPI, 
corroboration  is  obtained  through  method  requirements  for  the  collection  of  practice 
implementation  indicators  of  multiple  types  (see  “practice  implementation  indicator”). 

[SCAMPI  MDD] 

Data  Collection  Session  —  An  activity  during  which  information  that  will  later  be  used 
as  the  basis  for  observation  formulation  or  corroboration  is  gathered.  Data  collection 
sessions  (or  activities)  include  the  administration  and/or  analysis  of  instruments, 
document  review,  interviews,  and  presentations.  [ARCvi.i] 

Direct  Artifact  —  The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice  implementation.  May  be 
explicitly  stated  or  implied  by  the  practice  statement  or  associated  infonnative  material. 

[SCAMPI  MDD  method  overview] 


Discovery-Based  Appraisal  —  An  appraisal  in  which  limited  objective  evidence  is 
provided  by  the  appraised  organization  prior  to  the  appraisal,  and  the  appraisal  team  must 


probe  and  uncover  a  majority  of  the  objective  evidence  necessary  to  obtain  sufficient 
coverage  of  CMMI  model  practices.  Discovery-based  appraisals  typically  involve 
substantially  greater  appraisal  team  effort  than  verification-based  appraisals,  in  which 
much  of  the  objective  evidence  is  provided  by  the  appraised  organization.  (See 
verification-based  appraisal  for  contrast.)  [scampi  mdd] 

Document  —  A  collection  of  data,  regardless  of  the  medium  on  which  it  is  recorded,  that 
generally  has  pennanence  and  can  be  read  by  humans  or  machines,  [arc  vi.i]  In  SCAMPI, 
documents  are  work  products  reflecting  the  implementation  of  one  or  more  model 
practices.  This  typically  includes  work  products  such  as  organizational  policies, 
procedures,  and  implementation-level  work  products.  Documents  may  be  available  in 
hardcopy,  softcopy,  or  accessible  via  hyperlinks  in  a  web-based  environment.  [derived  from 

SCAMPI  MDD  method  overview] 


Indirect  Artifact  —  An  artifact  that  is  a  consequence  of  performing  a  specific  or  generic 
practice  or  that  substantiates  its  implementation,  but  that  is  not  the  purpose  for  which  the 
practice  is  perfonned.  This  indicator  type  is  especially  useful  when  there  may  be  doubts 
about  whether  the  intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but 
there  is  no  indication  of  where  it  came  from,  who  worked  to  develop  it,  or  how  it  is  used). 

[SCAMPI  MDD  method  overview] 


Instantiation  —  For  practices  implemented  by  projects,  each  project;  for  practices 
implemented  organization-wide,  the  instance,  [scampi  mdd] 

Instrument  —  Artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of  data 
(e.g.,  questionnaires,  organizational  unit  information  packets).  [ARCvi.qln  SCAMPI, 
instruments  are  used  to  collect  written  infonnation  relative  to  the  organizational  unit’s 
implementation  of  CMMI  model  practices.  This  can  include  assets  such  as 
questionnaires,  surveys,  or  an  organizational  mapping  of  CMMI  model  practices  to  its 
corresponding  processes,  [scampi  mdd] 

Interview  —  A  meeting  of  appraisal  team  members  with  appraisal  participants  for  the 
purpose  of  gathering  information  relative  to  work  processes  in  place,  [arc  vi.i]  In 
SCAMPI,  this  includes  face-to-face  interaction  with  those  implementing  or  using  the 
processes  within  the  organizational  unit.  Interviews  are  typically  held  with  various  groups 
or  individuals,  such  as  project  leaders,  managers,  and  practitioners.  A  combination  of 
formal  and  informal  interviews  may  be  held  and  interview  scripts  or  exploratory 
questions  developed  to  elicit  the  infonnation  needed,  [scampi  mdd] 

Mini-Team  —  A  subset  of  the  appraisal  team  members,  typically  two  or  three,  assigned 
primary  responsibility  for  collection  of  sufficient  appraisal  data  to  ensure  coverage  of 
their  assigned  reference  model  process  areas,  [scampi  mdd] 

Objective  Evidence  —  Qualitative  or  quantitative  information,  records,  or  statements  of 
fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence  and 
implementation  of  a  process  element,  which  is  based  on  observation,  measurement,  or 
test  and  which  can  be  verified,  [cmmi  model  glossary,  iso  98c  and  arc  vi.i]  In  SCAMPI,  sources  of 
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objective  evidence  include  instruments,  presentations,  documents,  and  interviews,  [scampi 

MDD] 

Observation  —  A  written  record  that  represents  the  appraisal  team  members’ 
understanding  of  information  either  seen  or  heard  during  the  appraisal  data  collection 
activities.  The  written  record  may  take  the  form  of  a  statement  or  may  take  alternative 
forms  as  long  as  the  infonnation  content  is  preserved,  [cmmi  model  glossary,  arc  vi.i] 

Organization’s  Set  of  Standard  Processes  (OSSP)  —  A  collection  of  definitions  of  the 
processes  that  guide  activities  in  an  organization.  These  process  descriptions  cover  the 
fundamental  process  elements  (and  their  relationships  to  each  other,  such  as  ordering  and 
interfaces)  that  must  be  incorporated  into  the  defined  processes  that  are  implemented  in 
projects  across  the  organization.  A  standard  process  enables  consistent  develop  [cmmi  model 

glossary] 


Organizational  Unit  (OU)  —  That  part  of  an  organization  that  is  the  subject  of  an 
appraisal  (also  known  as  the  organizational  scope  of  the  appraisal).  An  organizational 
unit  deploys  one  or  more  processes  that  have  a  coherent  process  context  and  operates 
within  a  coherent  set  of  business  objectives.  An  organizational  unit  is  typically  part  of  a 
larger  organization,  although  in  a  small  organization,  the  organizational  unit  may  be  the 

whole  Organization,  [derived  from  CMMI  model  glossary,  ISO  98C  and  ARC  vl.l] 


Practice  Characterization  —  The  assignment  of  a  value  describing  the  extent  to  which 
a  CMMI  model  practice  is  implemented,  used  as  a  mechanism  to  reach  appraisal  team 
consensus.  The  range  of  values  for  practice  characterization  values  includes  Fully 
Implemented  (FI),  Largely  Implemented  (LI),  Partially  Implemented  (PI),  and  Not 
Implemented  (NI).  Practice  characterization  values  are  assigned  to  each  CMMI  model 
practice  for  each  process  instantiation  within  the  appraisal  scope,  and  aggregated  to  the 
organizational  unit  level,  [local] 

Practice  Implementation  Indicators  (PIIs)  —  An  objective  attribute  or  characteristic 
used  as  a  “footprint”  to  verify  the  conduct  of  an  activity  or  implementation  of  a  CMMI 
model  specific  or  generic  practice.  Types  of  practice  implementation  indicators  include 
direct  artifacts,  indirect  artifacts,  and  affirmations.  [derived  from  15504-9  and  MDD  method  overview] 

Preliminary  Finding  —  Initial  findings  created  by  an  appraisal  team  after  consolidating 
and  synthesizing  valid  observations  to  provide  the  findings  to  appraisal  participants  for 
validation  of  accuracy.  [derived  from  ARC  vl.l] 

Presentation  —  In  SCAMPI,  a  source  of  objective  evidence  that  includes  information 
prepared  by  the  organization  and  delivered  visually  or  verbally  to  the  appraisal  team  to 
aid  in  understanding  the  organizational  processes  and  implementation  of  CMMI  model 
practices.  This  typically  includes  such  mechanisms  as  orientation  or  overview  briefings, 
and  demonstrations  of  tools  or  capabilities.  [derived  from  SCAMPI  MDD  method  overview] 


Process  Context  —  The  set  of  factors  documented  in  the  appraisal  input  that  influences 
the  judgment  and  comparability  of  appraisal  ratings.  These  include,  but  are  not  limited  to, 


(a)  the  size  of  the  organizational  unit  to  be  appraised,  (b)  the  demographics  of  the 
organizational  unit,  (c)  the  application  domain  of  the  products  or  services,  (d)  the  size, 
criticality,  and  complexity  of  the  products  or  services,  and  (e)  the  quality  characteristics  of 
the  products  or  services,  [cmmi  model  glossary] 

Process  Monitoring  —  An  appraisal  mode  of  usage  in  which  appraisals  are  used  to 
monitor  process  implementation  (for  example,  after  contract  award  by  serving  as  an  input 
for  an  incentive/award  fee  decision  or  a  risk  management  plan).  The  appraisal  results  are 
used  to  help  the  sponsoring  organization  tailor  its  contract  or  process  monitoring  efforts 
by  allowing  it  to  prioritize  efforts  based  on  the  observed  strengths  and  weaknesses  of  the 
organization’s  processes.  This  usage  mode  focuses  on  a  long-term  teaming  relationship 
between  the  sponsoring  organization  and  the  development  organization  (buyer  and 

supplier).  [derived  from  SCAMPI  MDD  method  overview] 


Relevant  Stakeholders  —  A  stakeholder  that  is  identified  for  involvement  in  specified 
activities  and  is  included  in  a  plan,  [cmmi  model  glossary] 

Strength  —  Exemplary  or  noteworthy  implementation  of  a  CMMI  model  practice,  [cmmi 

model  glossary  and  ARC  vl .  1  ] 


Sufficient  Data  Coverage  —  A  detennination  that  the  coverage  requirements  have  been 
met.  See  “coverage”  and  “coverage  criteria.”  [ARCvi.i] 

Supplier  Selection  —  An  appraisal  mode  of  usage  in  which  appraisal  results  are  used  as 
a  high  value  discriminator  to  select  suppliers.  The  results  are  used  in  characterizing  the 
process-related  risk  of  awarding  a  contract  to  a  supplier.  [derived  from  SCAMPI  MDD  method  overview] 

Valid  Observation  —  An  observation  that  the  appraisal  team  members  agree  is  (a) 
accurate,  (b)  corroborated,  and  (c)  consistent  with  other  valid  observations.  [ARCvi.i] 

Verification-Based  Appraisal  —  An  appraisal  in  which  the  focus  of  the  appraisal  team 
is  on  verifying  the  set  of  objective  evidence  provided  by  the  appraised  organization  in 
advance  of  the  appraisal,  in  order  to  reduce  the  amount  of  probing  and  discovery  of 
objective  evidence  during  the  appraisal  on-site  period.  (See  discovery-based  appraisal  for 
contrast.)  [scampi  mdd] 

Weakness  —  The  ineffective,  or  lack  of,  implementation  of  one  or  more  CMMI  model 

practices.  [CMMI  model  glossary  and  ARC  vl.  1] 
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